Quality is the ultimate goal. To succeed in this endeavor, olives are harvested at the greenish-yellow stage of ripeness (so, not really ripe at all) as early in the harvesting season as possible. At this fruit maturity, the oils have a high content of the healthy polyphenols, making them more bitter and pungent than a ripened olive. They also have a fresher, crisper, and cleaner flavor that is quite herbal.

However, the greener the olive, the less oil it contains, making it more difficult to extract. As the olives get slightly riper, to the fully red or slightly purple color stage, they have a lower content of polyphenols, so the oils come out less bitter and pungent. They taste more nutty, buttery, and tropical in flavor. In the absence of any defects, the ideal time to harvest the fruit really just comes down to personal preference and style.
– Paul Vossen