Project 6: Optimizing the Simulation
This is the second project on elephant population simulation. In the first project, you developed the overall simulation and used it to figure out a single parameter: the percentage of female elephants to dart each year. This week we're going to explore how to optimize one or more parameters of a simulation automatically. This will involve a little bit of restructuring of your elephant simulation code and then the development of an optimizer that will run the simulation many times in order to figure out parameters that best achieve a specific outcome.
Create a new file called optimize.py. Have the file import the sys,
elephant, and random packages. Then create a function
called optimize with the following definition.
def optimize( min, max, optfunc, simParameters = None, tolerance = 0.001, maxIterations = 20, verbose=False ): """Executes a search to bring the result of the function optfunc to zero. min: minimum parameter value to search max: maximum parameter value to search optfunc: function to optimize simParameters: optional parameter list to pass to optfunc tolerance: how close to zero to get before terminating the search maxIterations: how many iterations to run before terminating the search verbose: whether to print lots of information or not"""
The optimize function is very similar to the binary search function you wrote in lab. Start by assigning to a variable done the value False. Then, start a loop while done is False.
Inside the loop, first assign to testInput the average of max and min. This is not (should not be) an integer calculation. If verbose is True, print out testInput. Next, assign to result the return value of calling optfunc with testInput and simParameters as the arguments. If verbose is True, print out the result value.
Next, if the result is positive, assign to max the value of testInput. Else if the result is negative, assign to min the value of testInput. Else, assign to done the value True.
If max - min is less than the tolerance value, then assign to done the value True.
Decrement maxIterations. If maxIterations is less than or equal to zero, then set done to True.
Outside the loop, return testInput.
To test your optimize function, copy the following code and run optimize.py. As noted in the comments, try making tolerance smaller and see if it matches more digits in the target value.
def target(x, pars): """a function that returns x - target""" return x - 0.73542618 def testTarget(): """Tests the binary search using a simple target function. Try changing the tolerance to see how that affects the search.""" res = optimize( 0.0, 1.0, target, tolerance = 0.01, verbose=True) print res return if __name__ == "__main__": testTarget()
- The next step is to test the optimize function with your elephantSim function. Create a testEsim function (similar to testTarget above) that calls optimize with a min value of 0.0, a max value of 0.5, and passes it elephant.elephantSim as the target function. You probably want to set verbose=True as well. As with the testTarget function, assign the return value to a variable and then print the variable. At the bottom of your code, change testTarget() to testEsim() then run optimize.py. Does your optimize function find a value close to 0.43 for the percent darted?
The next step is to automate the process of evaluating the effects of
changing a simulation parameter across a range of values. This
function will let us discover, for example, the effect on the dart
percentage of changing the calfSurvival rate from 80% to 90% in steps
of 1%. The function definition is given below.
def evalParameterEffect( whichParameter, testmin, testmax, teststep, defaults=None, verbose=False ): """Evaluates the effects of the selected parameter on the dart percentage whichParameter: the index of the simulation parameter to test testmin: the minimum value to test testmax: the maximum value to test teststep: the step between parameter values to test defaults: default simulation parameters to use (default value of None)""" # if defaults is None, assign to simParameters the result of calling elephant.defaultSimParameters. # else, assign to simParameters a copy of defaults (e.g. simParameters = defaults[:] # create an empty list (e.g. results) to hold the results if verbose: print "Evaluating parameter %d from %.3f to %.3f with step %.3f" % (whichParameter, testmin, testmax, teststep) # assign to t the value testmin # while t is less than testmax # assign to the simParams[whichParameter] the value t # assign to percDart the result of calling optimize with the appropriate arguments, including simParameters # append to results the tuple (t, percDart) if verbose: print "%8.3f \t%8.3f" % (t, percDart) # increment t by the value teststep if verbose: print "Terminating" # return the list of results
Test your evalParameterEffects function by modifying your top level code at the bottom of your file to be the following. For the purposes of debugging, you will probably want to reduce the default carrying capacity to 1000 or even 500.
if __name__ == "__main__": evalParameterEffect( elephant.IDXProbAdultSurvival, 0.98, 1.0, 0.001, verbose=True )
What does this do? What should you expect the output to be?
Your final task is to make the following evaluations, showing the
effect on the dart percentage of the following parameter sweeps. Please use a carrying capacity of 2000. Make
a table or graph (or both) for each case. These five items should go
in your writeup.
- Vary the adult survival probability from 0.98 to 1.0 in steps of 0.001.
- Vary the calf survival probability from 0.80 to 0.90 in steps of 0.01.
- Vary the senior survival probability from 0.1 to 0.5 in steps of 0.05.
- Vary the calving interval from 3.0 to 3.4 in steps of 0.05.
- Vary the carrying capacity from 3500 to 7000 in steps of 500.
Your goal is to automate this process as much as possible. One option is to write a program that executes all of these in sequence and writes the results to a CSV file. Note that this option may take a while to test and debug. A second option is to make a program that runs only one of these at a time, but it lets you control it from the command line (specifying which parameter and the min/max/step). That is easier to debug, but you have to run it several times.
It is probably a good idea to figure out how to write data out to a CSV file and use that capability here.
fp = file( "filename here", "w") # open a file for writing fp.write( "write this string to the file\n") # write a string to the file fp.close() # close the file
Each assignment will have a set of suggested extensions. The required tasks constitute about 85% of the assignment, and if you do only the required tasks and do them well you will earn a B+. To earn a higher grade, you need to undertake one or more extensions. The difficulty and quality of the extension or extensions will determine your final grade for the assignment. One complex extension, done well, or 2-3 simple extensions are typical.
The following are a few suggestions on things you can do as extensions to this assignment. You are free to choose other extensions.
- Figure out how to automate the graphiing process using gnuplot.
- Have your program write out proper CSV files with a header line and appropriate commas.
- How much variation is there in the average total population for a 200-year elephant simulation across different runs? How stable is the estimate generated by doing 5 simulation runs?
- Enable the user to control your top level program with optional flags. For example, -par CarryingCapacity would specify that the program should evaluate carrying capacity, and -min 3500 would specify that it should start the evaluation at 3500.
- Check out the os package (import os). What could you do with the os.system function to automate your simulations?
Write-up and Hand-in
Turn in your code by putting it into your private hand-in directory on the Courses server. All files should be organized in a folder titled "Project 6" and you should include only those files necessary to run the program. We will grade all files turned in, so please do not turn in old, non-working, versions of files.
Make a new wiki page for your assignment. Put the label cs152s17project6 in the label field on the bottom of the page. But give the page a meaningful title (e.g. Milo's project 6).
In general, your intended audience for your write-up is your peers not in the class. Your goal should be to be able to use it to explain to friends what you accomplished in this project and to give them a sense of how you did it. Follow the outline below.
- A brief summary of the task, in your own words. This should be no more than a few sentences. Give the reader context and identify the key purpose of the assignment.
- A description of your solution to the tasks, including any text output or images you created. This should be a description of the form and functionality of your final code. Note any unique computational solutions you developed or any insights you gained from your code's output. You may want to incorporate code snippets in your description to point out relevant features. Code snippets should be small segments of code--usually less than a whole function--that demonstrate a particular concept. If you find yourself including more than 5-10 lines of code, it's probably not a snippet.
- A description of any extensions you undertook, including text output or images demonstrating those extensions. If you added any modules, functions, or other design components, note their structure and the algorithms you used.
- A brief description (1-3 sentences) of what you learned. Think about the answer to this question in terms of the stated purpose of the project. What are some specific things you had to learn or discover in order to complete the project?
- A list of people you worked with, including TAs and professors. Include in that list anyone whose code you may have seen, such as those of friends who have taken the course in a previous semester.
- Double-check the label. When you created the page, you should have added a the label cs152s17sproject6. Make sure it is there.
Thanks to Cathy Collins for the project idea and documentation. The original project concept and idea came from Therese Donovan, University of Vermont.