akxen / nemde Goto Github PK

Reverse engineering Australia's National Electricity Market Dispatch Engine (NEMDE)

License: Apache License 2.0

Python 98.57% Shell 0.83% Dockerfile 0.59%

nemde's Introduction

NEMDE Approximation

Australia's National Electricity Dispatch Engine (NEMDE) determines regional electricity prices and sets dispatch targets for generators and loads in Australia's National Electricity Market (NEM). This repository contains a model that seeks to approximate the operation of NEMDE. If you just want to interact with the model, and are less interested in model development, please see https://github.com/akxen/dispatch-api.

Key elements of the model's formulation have been inferred by analysing publicly available documents released by the Australian Energy Market Operator (AEMO). As NEMDE's mathematical formulation is not publicly available it is not possible to validate the approximated model's mathematical formulation directly. Instead, a data driven approach is used to evaluate the model's performance. This involves passing the approximate model of NEMDE historical case files describing the NEM's state, with the model using these parameters to formulate and solve a mathematical program. Outputs from the approximated model consist of prices and dispatch targets which are then compared with historical solutions reported by NEMDE. Close correspondence between solutions obtained from the approximate model and those reported by NEMDE indicates good model performance.

A Docker container is used run a MySQL database to store historical NEMDE case files and handle results outputted during validation runs. Use the following steps to test the model on your local machine.

Steps to evaluate model

Clone the repository:

git clone https://github.com/akxen/nemde.git

Setup MySQL container environment variables. Rename mysql/mysql-template.env to mysql/mysql.env and set MYSQL_PASSWORD and MYSQL_ROOT_PASSWORD variables.
Setup NEMDE container environment variables. Rename config/nemde-template.env to config/nemde.env and update entries. Ensure MYSQL_PASSWORD in config/nemde.env is the same as MYSQL_ROOT_PASSWORD specified in config/mysql.env.
Use casefiles/zipped/download_casefiles.sh to download historical NEMDE case files. The TEST_YEAR and TEST_MONTH variables within config/nemde.env should correspond to the monthly archive you have downloaded.
Run ./run_tests.sh to test the model using the settings specified within config/nemde.env. It may take a some time (1-2 hours) to build the containers and upload NEMDE case files in the MySQL database the first time you run this command. Logs are stored in ~/logs by default. Update run_tests.sh to change the output directory.

nemde's People

Contributors

Stargazers

Watchers

Forkers

jeremylloyd pkcool coryvegan nemde-dev

nemde's Issues

CurrentModeTime greater than defined limit

Issue observed for CaseID: 20191030149

From TraderCollection consider BRAEMAR1. The unit's CurrentMode is '1' and T1 is '1', but the CurrentModeTime is '5'. My understanding is that CurrentModeTime should be less than or equal to limit defined for that mode.

{
            "@TraderID": "BRAEMAR1",
            "@TraderType": "GENERATOR",
            "@FastStart": "1",
            "@MinLoadingMW": "110",
            "@CurrentMode": "1",
            "@CurrentModeTime": "5",
            "@T1": "1",
            "@T2": "1",
            "@T3": "1",
            "@T4": "1",
            "@SemiDispatch": "0",
            "TraderInitialConditionCollection": {
              "TraderInitialCondition": [
                {
                  "@InitialConditionID": "AGCStatus",
                  "@Value": "0"
                },
                {
                  "@InitialConditionID": "HMW",
                  "@Value": "146.699996948242"
                },
                {
                  "@InitialConditionID": "InitialMW",
                  "@Value": "0"
                },
                {
                  "@InitialConditionID": "LMW",
                  "@Value": "5.99949359893799"
                },
                {
                  "@InitialConditionID": "SCADARampDnRate",
                  "@Value": "720.021572113037"
                },
                {
                  "@InitialConditionID": "SCADARampUpRate",
                  "@Value": "720.021572113037"
                },
                {
                  "@InitialConditionID": "WhatIfInitialMW",
                  "@Value": "0"
                }
              ]
            },

Perhaps a unit can stay in a given mode for longer than the specified duration i.e. it is a strict lower bound, but not necessarily an upper bound on the mode time. Need to investigate further and perhaps reformulate the fast-start unit inflexibility profile constraints. Will not touch for now though - can do this when implementing fast-start unit multi-solve approach. Need to specify this as a caveat for users.

Allocated MNSP losses

Issue observed for CaseID: 20191022235

To-date I have assumed MNSP InitialMW determines how losses are allocated at the end of the dispatch interval. There seem to be problems with this approach for some dispatch intervals. I think this is due to the direction of flow changing over the course of the dispatch interval. E.g. if InitialMW is negative at the start of the dispatch interval, but positive at the end of the dispatch interval, then the E_REGION_MNSP_LOSS is incorrect - losses are allocated to the wrong node. I suspect NEMDE does more than one run to handle this case. If the direction of interconnector flow solution is of a different sign to InitialMW, then a second run is performed to allocate losses to the sending node.

For now this limitation will be a caveat made known to the user. In the future a multi-solve framework could be used to fix this issue.

CurrentMode=0 and EnergyTarget > 0

Following the "Fast-Start Inflexibility profile.pdf" document, units with CurrentMode=0 can be committed for dispatch. The process involves two NEMDE runs. From the docs:

"A fast-start commitment solve is performed in the first pass. In this solve all fast-start units are modelled with their dispatch inflexibility profile constraints ignored. Any bid and SCADA-metered energy ramp rate constraints and joint ramping constraints for fast-start units that are in modes 0, 1, or 2 at the start of the current dispatch interval are also ignored."

Also from the docs:

"Units in Mode 0 at the start of the current dispatch interval are candidates for start-up (entering into Mode 1). A unit will be started if its MW result from Step 1 is greater than the fast-start threshold of 0.005 MW. The target mode for the end of current dispatch interval, which is used in Step 3, is calculated according to its start-up profile. For example, units having T1+T2 < 5 minutes will enter into Mode 3. Step 3 may then determine non-zero MW generation targets for the end of the current dispatch period."

In order to correctly implement fast-start inflexibility profile constraints a multi-solve framework must be adopted. However, there is a trade-off between model accurracy and solve time. To implement this approach would require 3 solves instead of the current 2 (at present one is used to determine dispatch targets, and another used to determine prices).

This deficiency can be listed as a caveat for users. In the future a multi-solve framework incorporating inflexibility profile solves could be adopted, with the user able to choose which model to use. For now I'm not going to incorporate this feature as the cost of the additional complexity seems to outweigh the benefit - numerical experiments suggest this issue has a material impact only in a small percentage of dispatch intervals.

Missing offer type in generic constraint

Case ID: 20210405178

APD01 has L5MI variable in generic constraint, but not L5MI bid. This results in a KeyError being raised when constructing constraints linking generic constraint trader variables to total offer variables for each trade type.

Price bands:

<Trader TraderID="APD01" TraderType="NORMALLY_ON_LOAD" SemiDispatch="0">
<TraderInitialConditionCollection>
	<TraderInitialCondition InitialConditionID="AGCStatus" Value="0" />
	<TraderInitialCondition InitialConditionID="InitialMW" Value="0" />
	<TraderInitialCondition InitialConditionID="WhatIfInitialMW" Value="0" />
</TraderInitialConditionCollection>
<TradePriceStructureCollection>
	<TradePriceStructure TradePriceStructureID="20210405178">
		<TradeTypePriceStructureCollection>
			<TradeTypePriceStructure TradeType="R5MI" PriceBand1="0.26" PriceBand2="0.75" PriceBand3="1.49" PriceBand4="2.29" PriceBand5="2.77" PriceBand6="13.75" PriceBand7="18.89" PriceBand8="22.99" PriceBand9="2399" PriceBand10="12500" Offer_SettlementDate="2021-04-05T00:00:00+10:00" Offer_EffectiveDate="2021-04-05T18:40:44+10:00" Offer_VersionNo="1" />
			<TradeTypePriceStructure TradeType="R60S" PriceBand1="0.26" PriceBand2="0.75" PriceBand3="1.49" PriceBand4="2.29" PriceBand5="2.77" PriceBand6="13.75" PriceBand7="18.89" PriceBand8="22.99" PriceBand9="2399" PriceBand10="12500" Offer_SettlementDate="2021-04-05T00:00:00+10:00" Offer_EffectiveDate="2021-04-05T18:40:30+10:00" Offer_VersionNo="1" />
			<TradeTypePriceStructure TradeType="R6SE" PriceBand1="0.25" PriceBand2="0.5" PriceBand3="1.49" PriceBand4="2.29" PriceBand5="2.77" PriceBand6="13.75" PriceBand7="18.89" PriceBand8="50" PriceBand9="2399" PriceBand10="12500" Offer_SettlementDate="2021-04-05T00:00:00+10:00" Offer_EffectiveDate="2021-04-05T18:40:18+10:00" Offer_VersionNo="1" />
		</TradeTypePriceStructureCollection>
	</TradePriceStructure>
</TradePriceStructureCollection>
</Trader>

Quantity bands:

<TraderPeriod TraderID="APD01" RegionID="VIC1" TradePriceStructureID="20210405178">
<TradeCollection>
	<Trade TradeType="R5MI" MaxAvail="450" EnablementMin="0" EnablementMax="0" LowBreakpoint="0" HighBreakpoint="0" BandAvail1="30" BandAvail2="15" BandAvail3="20" BandAvail4="20" BandAvail5="20" BandAvail6="20" BandAvail7="10" BandAvail8="0" BandAvail9="0" BandAvail10="315" />
	<Trade TradeType="R60S" MaxAvail="450" EnablementMin="0" EnablementMax="0" LowBreakpoint="0" HighBreakpoint="0" BandAvail1="30" BandAvail2="10" BandAvail3="10" BandAvail4="20" BandAvail5="20" BandAvail6="20" BandAvail7="15" BandAvail8="10" BandAvail9="0" BandAvail10="315" />
	<Trade TradeType="R6SE" MaxAvail="450" EnablementMin="0" EnablementMax="0" LowBreakpoint="0" HighBreakpoint="0" BandAvail1="31" BandAvail2="0" BandAvail3="0" BandAvail4="0" BandAvail5="0" BandAvail6="0" BandAvail7="0" BandAvail8="0" BandAvail9="0" BandAvail10="419" />
</TradeCollection>
</TraderPeriod>

Generic constraint:

<GenericConstraint ConstraintID="F_V+NIL_APD01_L5" Version="20130823000000_1" EffectiveDate="2013-08-23T00:00:00+10:00" VersionNo="1" Type="LE" ViolationPrice="17400000" RHS="0" Force_SCADA="False">
<LHSFactorCollection>
	<TraderFactor Factor="1" TradeType="L5MI" TraderID="APD01" />
</LHSFactorCollection>
<RHSTermCollection>
	<RHSTerm TermID="1" Multiplier="1" SpdID="APD01_LOAD" SpdType="A" Default="0" />
	<RHSTerm TermID="2" Multiplier="1" SpdID="APD02_LOAD" SpdType="A" Default="0" />
	<RHSTerm TermID="3" Multiplier="-15" SpdID="Constant" SpdType="C" Default="0" />
	<RHSTerm TermID="4" Multiplier="10000" Operation="STEP" SpdID="Swamp" SpdType="U" Default="0" />
</RHSTermCollection>
<s:ConstraintTrkCollection xmlns:s="http://www.w3.org/2001/XMLSchema-instance">
	<ConstraintTrkItem Invocation_ID="50180" Start_Interval_DateTime="2021-04-05T18:50:00+10:00" End_Interval_DateTime="2021-04-05T18:50:00+10:00" DynamicRHS="1" GenConSetID="F-I_NIL" Intervention="False" ASConstraintType="NETWORK" SystemNormal="True" Invocation_Group_ID="50180" LimitType="FCAS" />
</s:ConstraintTrkCollection>
</GenericConstraint>

Proposed solution is to skip linking constraints if key error is raised.

MNSP constraint violation

Need to add constraint violation variables to MNSP loss model. Not sure to what constraints these variables should apply. Also need to include constraint violation penalty in model's objective function.

Missing CurrentModeTime

Case ID: 20210419241

Both ADPBA1G and ADPBA1L missing CurrentModeTime parameter. Must handle this case when parsing case file.

<Trader TraderID="ADPBA1G" TraderType="GENERATOR" FastStart="1" MinLoadingMW="0" T1="0" T2="0" T3="0" T4="0" SemiDispatch="0">
<TraderInitialConditionCollection>
	<TraderInitialCondition InitialConditionID="AGCStatus" Value="0" />
	<TraderInitialCondition InitialConditionID="InitialMW" Value="0" />
	<TraderInitialCondition InitialConditionID="WhatIfInitialMW" Value="0" />
</TraderInitialConditionCollection>
<TradePriceStructureCollection>
	<TradePriceStructure TradePriceStructureID="20210419241">
		<TradeTypePriceStructureCollection>
			<TradeTypePriceStructure TradeType="ENOF" PriceBand1="0" PriceBand2="0.99" PriceBand3="1.97" PriceBand4="3.95" PriceBand5="7.89" PriceBand6="15.78" PriceBand7="31.56" PriceBand8="63.13" PriceBand9="126.25" PriceBand10="252.51" Offer_SettlementDate="2021-04-19T00:00:00+10:00" Offer_EffectiveDate="2021-04-16T15:29:37+10:00" Offer_VersionNo="1" />
		</TradeTypePriceStructureCollection>
	</TradePriceStructure>
</TradePriceStructureCollection>
</Trader>
<Trader TraderID="ADPBA1L" TraderType="LOAD" FastStart="1" MinLoadingMW="0" T1="0" T2="0" T3="0" T4="0" SemiDispatch="0">
<TraderInitialConditionCollection>
	<TraderInitialCondition InitialConditionID="AGCStatus" Value="0" />
	<TraderInitialCondition InitialConditionID="InitialMW" Value="0" />
	<TraderInitialCondition InitialConditionID="WhatIfInitialMW" Value="0" />
</TraderInitialConditionCollection>
<TradePriceStructureCollection>
	<TradePriceStructure TradePriceStructureID="20210419241">
		<TradeTypePriceStructureCollection>
			<TradeTypePriceStructure TradeType="LDOF" PriceBand1="0" PriceBand2="0.99" PriceBand3="1.97" PriceBand4="3.95" PriceBand5="7.89" PriceBand6="15.78" PriceBand7="31.56" PriceBand8="63.13" PriceBand9="126.25" PriceBand10="252.51" Offer_SettlementDate="2021-04-19T00:00:00+10:00" Offer_EffectiveDate="2021-04-16T15:29:37+10:00" Offer_VersionNo="1" />
		</TradeTypePriceStructureCollection>
	</TradePriceStructure>
</TradePriceStructureCollection>
</Trader>

Proposed solution is to set CurrentModeTime to 0 if not found. This seems reasonable given the startup profile times are all 0 - fast start storage unit will come online 'instantly'.

API data format for user supplied model parameters

The idea is to adopt a functional approach to application of user supplied model data. First, a case ID is specified which serves as the basis. User supplied model parameters then update these basis parameters to generate a case to be run. Need to determine the data format that should be used when running this case.

There are two options: closely follow the NEMDE data structure which is a bit clumsly and complicated, or develop a new format that simplifies the process. The first is probably best for those familiar with NEMDE, the second better for those new to the platform. I think the second option is best here - the goal is get people to use the software so it should be as intuitive as possible.

Data categories:

Case
Traders
Interconnectors
Regions
Constraints

Traders:

(TraderID):
     Info: {
            "@TraderID": "AGLHAL",
            "@TraderType": "GENERATOR",
            "@FastStart": "1",
            "@MinLoadingMW": "2",
            "@CurrentMode": "0",
            "@CurrentModeTime": "0",
            "@T1": "1",
            "@T2": "3",
            "@T3": "10",
            "@T4": "2",
            "@SemiDispatch": "0",
}

Ramp rates for fast-start units

NEMDE Case ID: 20191008021

Trader dispatch targets should be constrained by SCADA ramp rates / energy offer ramp rates. For URANQ13 it's hard to tell what ramp-rate has been used to constrain dispatch. Consider the following inputs:

From TraderCollection:

{
            "@TraderID": "URANQ13",
            "@TraderType": "GENERATOR",
            "@FastStart": "1",
            "@MinLoadingMW": "80",
            "@CurrentMode": "2",
            "@WhatIfCurrentMode": "2",
            "@CurrentModeTime": "4",
            "@WhatIfCurrentModeTime": "4",
            "@T1": "6",
            "@T2": "7",
            "@T3": "39",
            "@T4": "7",
            "@SemiDispatch": "0",
            "TraderInitialConditionCollection": {
              "TraderInitialCondition": [
                {
                  "@InitialConditionID": "AGCStatus",
                  "@Value": "0"
                },
                {
                  "@InitialConditionID": "HMW",
                  "@Value": "173"
                },
                {
                  "@InitialConditionID": "InitialMW",
                  "@Value": "23.6062469482422"
                },
                {
                  "@InitialConditionID": "LMW",
                  "@Value": "79.9937515258789"
                },
                {
                  "@InitialConditionID": "SCADARampDnRate",
                  "@Value": "659.906272888184"
                },
                {
                  "@InitialConditionID": "SCADARampUpRate",
                  "@Value": "659.906272888184"
                },
                {
                  "@InitialConditionID": "WhatIfInitialMW",
                  "@Value": "45.71429"
                }
              ]
            }

From TraderPeriodCollection:

{
                "@TraderID": "URANQ13",
                "@RegionID": "NSW1",
                "@TradePriceStructureID": "20191008021",
                "TradeCollection": {
                  "Trade": {
                    "@TradeType": "ENOF",
                    "@RampUpRate": "660",
                    "@RampDnRate": "660",
                    "@MaxAvail": "166",
                    "@BandAvail1": "80",
                    "@BandAvail2": "166",
                    "@BandAvail3": "0",
                    "@BandAvail4": "0",
                    "@BandAvail5": "0",
                    "@BandAvail6": "0",
                    "@BandAvail7": "0",
                    "@BandAvail8": "0",
                    "@BandAvail9": "0",
                    "@BandAvail10": "0"
                  }
                }
              },

From TraderSolution (I am primarily focused on the physical run, hence @Intervention = "1":

        {
          "@TraderID": "URANQ13",
          "@PeriodID": "2019-10-08T05:45:00+10:00",
          "@Intervention": "1",
          "@EnergyTarget": "101.99688",
          "@R6Target": "0",
          "@R60Target": "0",
          "@R5Target": "0",
          "@R5RegTarget": "0",
          "@L6Target": "0",
          "@L60Target": "0",
          "@L5Target": "0",
          "@L5RegTarget": "0",
          "@R6Price": "0",
          "@R60Price": "0",
          "@R5Price": "0",
          "@R5RegPrice": "0",
          "@L6Price": "0",
          "@L60Price": "0",
          "@L5Price": "0",
          "@L5RegPrice": "0",
          "@R6Violation": "0",
          "@R60Violation": "0",
          "@R5Violation": "0",
          "@R5RegViolation": "0",
          "@L6Violation": "0",
          "@L60Violation": "0",
          "@L5Violation": "0",
          "@L5RegViolation": "0",
          "@FSTargetMode": "3",
          "@RampUpRate": "659.91",
          "@RampDnRate": "659.91",
          "@RampPrice": "-58.77",
          "@RampDeficit": "0"
        },

From TraderSolution (for good measure I have included results from the pricing run):

        {
          "@TraderID": "URANQ13",
          "@PeriodID": "2019-10-08T05:45:00+10:00",
          "@Intervention": "0",
          "@EnergyTarget": "101.99688",
          "@R6Target": "0",
          "@R60Target": "0",
          "@R5Target": "0",
          "@R5RegTarget": "0",
          "@L6Target": "0",
          "@L60Target": "0",
          "@L5Target": "0",
          "@L5RegTarget": "0",
          "@R6Price": "0",
          "@R60Price": "0",
          "@R5Price": "0",
          "@R5RegPrice": "0",
          "@L6Price": "0",
          "@L60Price": "0",
          "@L5Price": "0",
          "@L5RegPrice": "0",
          "@R6Violation": "0",
          "@R60Violation": "0",
          "@R5Violation": "0",
          "@R5RegViolation": "0",
          "@L6Violation": "0",
          "@L60Violation": "0",
          "@L5Violation": "0",
          "@L5RegViolation": "0",
          "@FSTargetMode": "3",
          "@RampUpRate": "659.91",
          "@RampDnRate": "659.91",
          "@RampPrice": "-93.94",
          "@RampDeficit": "0"
        },

Note how InitialMW=23.6 and the ramp-rate (given by SCADA and in the energy offer is approximately 660 MW/hr). Therefore, the unit's ramping capability over a 5min interval is 660/12=55MW, so the ramping up constraint will ensure @EnergyTarget <= 23.6 + 55 = 78.6. But note the energy target is 101.99688MW. The unit's output seems to exceed its ramping capability.

Note that this unit is initially operating on the T2 section of its fast-start inflexibility profile - it is ramping up to a min loading of 80MW. The ramp-rate over this interval is 80/7 = 11.43 MW/min = 685.7 MW/hr. But even this ramp rate is insufficient to meet the energy target (23.6 + (11.43x5) = 80.75 MW).

The unit also seems to have deviated from its fixed inflexibility profile while in T2. The unit has been in T2 for 4 mins, so it should have an output of (80/7) x 4 = 45.7142 MW, but its InitialMW is only 23.6MW. The expected MW output corresponds to the WhatIfInitialMW value, so it's not clear if this value is used instead. Even if this value is used instead this still doesn't resolve the issue. Applying the SCADA ramp rate yields a max energy target of 45.7142 + (659.91/12) = 100.7067 MW which is still below the target. Using the startup profile ramp rate yields an max energy target of 45.7142 + (685.7/12) = 102.855 MW which is now above the target. I would expect the ramp rate constraint be binding in this scenario.

Some NEMDE case files missing

Some NEMDE case files are missing e.g. NEMSPDOutputs_2019103109300.loaded. Will need notify user of this.

Model objective value < observed objective value

While there is close correspondence between the model and observed objective values, the model objective value is consistently $20-30 less than the observed value. I think this is due to the tie-breaking model. Deviations in model energy targets and observed energy targets are also observed at times, but the objective values remain closes to the observed values.

Must re-think how tie-breaking model has been implemented in NEMDE. Goal is to identify a model that closes the gap between the model and observed objective values.