Climate change is one of humanity's major challenges, impacting our society, ecosystems, health, and economy. Human progress has relied on harnessing increasingly dense energy sources: from wood to coal, oil, and, most recently, nuclear fission. However, this reliance on fossil fuels has come at a cost. Burning these fuels releases greenhouse gases, especially carbon dioxide (CO$_2$), into the atmosphere, leading to a range of detrimental effects. The most significant of these is the steady rise in global average temperatures since the 1960s, resulting in unpredictable weather patterns and extreme events like tornadoes, wildfires, droughts, and floods.

Artificial Intelligence (AI) is emerging as a powerful tool to address climate change challenges. AI's capacity to analyze vast amounts of data, predict outcomes, and optimize resources makes it invaluable in this fight. At Tryolabs, we leverage our AI expertise to develop solutions that address diverse problems within the broader AI for Social Good (AI4SG) space.

In this blog post, we will share our collaboration with Open Climate Fix (OCF), a British non-profit organization dedicated to reducing greenhouse gas emissions through technical solutions. Our joint efforts focus on improving solar panel generation prediction, contributing to the broader goal of enhancing renewable energy solutions for a more sustainable future.

Figure 1: Solar radiation pathways to photovoltaic panels

The complexity of energy management

The electricity we rely on comes from a mix of sources, such as coal, natural gas, nuclear, and renewable power plants, all of which together make up the electricity grid. The mix of these sources varies by geographical location and local energy policies. Some countries, like the United States, rely mostly on fossil fuels, while others such as Uruguay, have a predominantly green energy matrix.

Managing the electricity grid requires a complex balance of matching supply with demand in real time to maintain stability. Public utilities across various countries use advanced forecasting methods and maintain capacity reserves to ensure a consistent electricity supply. These forecasts become more accurate as the time of demand approaches. Initially, forecasts cover broader timeframes, like an entire day, with a higher margin of error. However, as the specific time of demand draws nearer, predictions become more precise, enhancing grid reliability and reducing the risk of power outages.

Integrating renewable energy sources, like solar panels, adds complexity due to their fluctuating output, influenced by weather conditions and sunlight availability. Therefore, accurately predicting solar energy production is vital for seamlessly incorporating solar power into the grid.

Improving the precision of renewable energy forecasts enables grid operators to rely more on renewable sources and reduce the need for fossil fuel-based power. This allows for more efficient planning and management of energy resources, reduces operational costs associated with keeping fossil fuel generators on standby, and contributes to a more efficient and sustainable power grid. Precise predictions are essential for the successful transition to renewable energy systems.

Understanding these energy management intricacies highlights the need for improved forecasting. Keep reading to discover how AI can revolutionize the prediction and integration of solar energy into the grid.

Our collaboration: optimizing solar energy

Open Climate Fix's purpose

OCF tackles climate change by using open-source data and Machine Learning to create practical solutions. Their mission is to reduce greenhouse gas emissions and accelerate the transition to renewable energy. OCF focuses on collaborative projects, leveraging data and AI to improve forecasting and efficiency in areas such as solar energy generation, energy storage, and grid management. By making their research and tools open-source, they aim to foster innovation and enable widespread adoption of sustainable practices. Harnessing cutting-edge technologies like AI, OCF strives to improve the accuracy of energy predictions, ultimately leading to a more sustainable and efficient energy future.

The challenge

One of OCF's latest projects tackles the complex challenge of using AI to forecast the energy output of solar panels up to 48 hours in advance. This project aims to create an open-source tool for predicting solar energy that is accessible and beneficial to all. It leverages data on the panel's location, specific characteristics, and various meteorological variables to achieve this.

Predicting the energy output of a photovoltaic panel is a multifaceted challenge. Weather conditions, such as temperature and cloud cover, significantly impact the panel's energy generation. Additionally, accurately mapping the location and tilt of existing solar installations can be difficult, as many installations are done by private individuals or companies who are not obligated to report their setups.

Given these inherent challenges, enhancing the accuracy of solar output predictions is hard. Over the past few months, we have collaborated closely with OCF to improve a public model they have available. This partnership has led to significant advances in predicting solar energy production using public data.

A walk through our partnership

During our collaboration, we worked on training a new model to increase prediction reliability for solar panel generation based on panel information and weather data. OCF already had a publicly available model that users could access for these predictions. Our goal was to enhance this model while ensuring that the developed code and the obtained model remained open source. This approach promotes transparency, fosters community collaboration, and facilitates widespread adoption and improvement of the technology within the renewable energy sector.

Panel metadata and solar power generation ground truth were obtained from UK PV dataset, while weather data was retrieved from Open-Meteo, an open-source weather API. The complete set of meteorological variables used, along with their descriptions, is shown in the table below.

The Open-Meteo API provides data up to the present in hourly resolution, while the solar panel energy generation data in UK PV dataset spans from 2018 to 2021. Therefore, our study was restricted to this period. The data underwent careful analysis and preparation, including the removal of outliers such as nighttime solar energy production, extended periods of zero measurements, and panels with significantly deviant energy distributions.

To boost performance, we replaced the previous Gradient Boosting model with an XGBoost model; both based on ensemble learning using Decision Trees. The algorithm builds a sequential series of shallow Decision Trees, with each tree aiming to correct the errors of its predecessor, resulting in a model with high predictive skill. While both models share a foundational algorithmic approach, XGBoost offers distinct advantages over conventional Gradient Boosting. It is designed to be extremely fast and scalable, making it well-suited for handling large datasets. Additionally, it includes built-in regularization techniques that help mitigate overfitting, further enhancing its effectiveness.

The new model was trained and validated on data from 1,147 solar power panels, with testing on an additional 37 panels that were excluded from the training set. All panels are located in the United Kingdom. Detailed hyperparameter tuning using Optuna was performed, reducing prediction error from 0.19kW to 0.12kW on the test set, which is an improvement of 37% compared to the original model. Figure 2 illustrates the locations and errors of the 37 test panels.

Figure 2: Spatial distribution of the errors in the predictions.

Figure 3 shows an example of a 48-hour time frame prediction for one solar panel compared to the actual observations. The predictions of both models, the Gradient Boosting model and the XGBoost model, are displayed. In this plot the Gradient Boosting model clearly underestimates the observations. However, it is important to note that this plot represents only one example and is not indicative of all predictions.

Figure 3: Example prediction compared to the true observations for one solar panel for a 48-hour time frame.

In addition to improving accuracy, the new model significantly reduced memory and compute requirements by enabling advanced parameters in XGBoost's implementation. For example, optimizing the training process of individual Decision Trees reduced the number of necessary splits and decreased training time. Further details on these parameters can be found in the XGBoost documentation. This reduction in computational resources made training on less powerful computers feasible, enhancing project accessibility.

The developed model allows for predicting the solar energy output of specific panels up to 48 hours in advance, given a location and panel metadata. A schematic overview of the application can be seen in Figure 4.

Figure 4: Schematic sketch of the solar power predictions showing the user input needed, the automated prediction performed by the API and an example output.

The data and code used to train this model are entirely open-source, allowing researchers, developers, and enthusiasts to contribute to further improving solar energy prediction. The model was trained and evaluated using data from solar power panels in the United Kingdom. Although the model can be applied globally, caution is advised when using it outside the training dataset’s geographical bounds, as different regions may exhibit unique patterns in solar radiation, weather, and panel technology, which can affect prediction accuracy. We invite the global community to participate in this project by contributing data from other geographical areas, testing the model in new contexts, or even developing new applications.

Currently, there is no easy-to-use web application available, so the model must be run in Python. An example of how to install all necessary dependencies and apply the model is provided in this Colab notebook.

AI for a sustainable future

Our partnership with OCF exemplifies the potential of AI in combating climate change. By leveraging open-source technology, we have made advanced predictive models for solar energy production accessible to a broader audience.

Explore our GitHub solar forecast training repository to see how we processed the data and trained the models. You can even use this repository to train a model with your own data. Additionally, check out OCF's GitHub repository to see the models in action and access to a Colab notebook for hands-on experimentation.

If you own a solar panel, you can use these models to generate your own solar forecast. Need help? Don’t hesitate to reach out!

At Tryolabs, we leverage our AI expertise to tackle various global challenges. We have collaborated with UNICEF to identify impoverished schools and with the Lion Identification Network of Collaborators (LINC) to track lions using computer vision. Our work with OCF is just one example of how AI can drive positive change. By continuing to innovate and share our knowledge, we strengthen our collective ability to face the critical challenge of climate change.