Question:
There is a server with 1gb RAM. There is a log file nginx (any other web server) on 70gb. How to collect statistics on the user agent of site users as quickly as possible, taking into account the described resource restrictions.
Answer:
You read the file line by line and analyze the read term. In almost all languages, you can specify the buffer size when working with the file system, and therefore, despite the fact that you work with the file line by line, the system will read the file in fixed blocks.