https://simontoth.substack.com/p/daily-bite-of-c-optimizing-code-to

Very interesting...

But is the extra coding effort worth it? Real world performance should be 'good enough' - but that begs the question 'what is good enough' in any particular context?

The author's conclusions at the end of the article pretty much sum up what you mention, seeplus. :)

There's optimizations that benefit code to make it speedier, and then there's "optimizations!" Lots of time and skull sweat expended that end up with diminishing returns.

Are the customers/users gonna notice the time difference beyond basic optimizations? Probably not.

Best summarised as
- taking days of effort
- to save seconds of time
- on something run maybe once a week at most.

Except now it's a maintenance nightmare of "WTF is this doing?".

Anyone any ideas as to how to create the used 1B lines 12GB test file measurements.txt on Windows without using Java?

Level of optimization always depends on need. For example, the stuff that modern web browsers do to load and display a page quickly is some pretty aggressive, non-obvious, and non-trivial micro optimizations. But in the case of web browsers, that’s the difference between someone using your browser or a competing browser.

Knowing how to find and optimize stuff is both fun and useful, and an excellent way to improve your skills with a personal project every now and then.

seeplus wrote:
Anyone any ideas as to how to create the used 1B lines 12GB test file measurements.txt on Windows without using Java?

Just write a little program to do it. You can even randomize the content so that each line is unique. Shouldn’t cost more than 50–60 lines.

there are a ton of places where things take a while to process. Modern sensors produce more data than can be processed in real time without serious hardware or good optimizations. Just a standard image off a standard digital camera now has many millions of pixels, each with 3, sometimes 4 values and when you go to do fancy stuff over that (ray traces or 5x5 or bigger filters and so on) it can get sluggish if you don't do it right. And dealing with high rez lidar etc is just intense. Its great to have folks write up the basics every now and again, as the language tools/details change even if the concepts rarely do.

What gets me is the general *acceptance* of sluggish stuff. I have probably said that before but watching people sit there and wait for something to cook without being upset by it boggles my mind. My 5 years in database work ... a meta team of maybe 50 people broken into smaller teams of 10 or so... and hardly a ONE of them (other than myself) felt it was inappropriate to have data movement jobs that ran for most of a weekend. And this was maybe 75 million records, not billions. And they had zero interest in improvement... they thought it was normal and acceptable. Same for user level stuff too... computer throws up that windows ring of 'wait for nothing' and people just alt tab or watch it spin, but rarely complain.

I’m increasingly impressed with how bloat fills available resources.

A while ago I was reading about multi-threading, the author basically said: "Prefer to use an STL algorithm to do it over writing your own threaded code, if possible". I wonder this example could not have used the for_each STL algorithm, which can multi thread and even possibly vectorise the processing of the data.

An upcoming wee project for me is to have a go at fooling around with CUDA programming. A few years ago I had code that transforms grid coordinates into Lat/Long using the GeographicLib written Dr Charles Karney. My computer then was an average i7, IIRC it took about 0.96 seconds to process 1 million points on a single thread. The library does not seem to be multi threaded. My current machine i9 has about 3,300 cores in the NVidia GPU, interesting to see how quick it can do the same problem.

I have also read about Big Data, with the 3 V's: Volume (number of records); Velocity (transactions per second); Variability (non-uniformity of stored data). In terms of speed, I guess it's the same old problem: some people are happy waiting 1 second for something to complete; others need it as quick as possible, maybe max 1 milliseconds.

There’s an entire section in the text about using threads.

Duthomhas wrote:
There’s an entire section in the text about using threads.

Yes, but I am saying perhaps one could use for_each instead, and I should have said with execution policy :

https://en.cppreference.com/w/cpp/algorithm/execution_policy_tag_t

Seems to me so much easier to use, than writing one's own threads, unless there is a specific need.

Perhaps there is a reason?

Ah, I misread.

IDK. Good question.