The tool generates a number of downstream outputs and these include synthesizable RTL of logic designs and its appropriate test benches and different standard formats. The generated synthesizable logic RTL should be checked for compilation and synthesis error. Also with any change in the specification of the input spec a different view of the downstream output is formed. Each of the downstream outputs created from the same spec need to be checked for its error.Testing such a tool is a tedious and demanding task and a major part of the time is dedicated to the analysis of the status of the test. We have a regression environment which consists of hundreds of tests which verify the transformation of the tool to a latest version from a previous one. We use our own Verification planning and management tool IVerifySpec for the verification of IDesignSpec. This is the case of Doctor eating his own medicine since we created IVerifySpec for the sole purpose of ensuring that a product is verified to meet its specs. What better way to prove that it does so, than to use IVerifySpec to verify IDesignSpec. IVSExcel is a plugin for Microsoft Excel that enables a verification team to plan and analyze verification progress right inside Excel. A pair of XML file for verification is generated when the regression is run, one for the plan and the other for displaying the results of the tests. A Plan for IDS Regression is created in IVSExcel by importing the plan XML file into the plugin and the status of the Regression can be brought into IVS by annotating the result file. When a regression is run for testing the next version of the tool, all the changes with respect to the previous version can be brought into IVSExcel again through import and annotate of XML files but in an update mode. We were able to easily detect the trends and issues in the IDS regression in IVSExcel with the features that highlights the updates in the regression and also by filtering of test status with the help of properties and expressions involving properties. Here are some of the features that IVSExcel provides and their use to the end user. Table 1: IVS Features
|Create verification plan from Simulation database||User can either create plan in Excel by identifying the key metric used or create the plan from the simulation results from any vendor database.|
|Update verification plan with coverage reports and test reports.||Not only coverage reports can be annotated onto the plan, but also test results can be annotated.|
|The annotation can be done either as a new dataset or a merge into an existing dataset.||A merge, as opposed to new data set, is very useful in seeing what has changed compared to the last dataset. This is extremely useful for regression analysis.|
|Update verification plan and results||The same plan can be updated, this eliminates the need to create new Excel files, which in turn helps manage the large amount of data generated in verification.|
|Highlights changes in Plan and results in update mode||Easy detection of trends and issues in verification process.|
|Message display and capability to view log files from within the tool||Error messages from live regression and simulation environment can easily be displayed in the plan.Links to log files are provided enabling user to refer to the source files, log files, html pages of Coverage reports etc.|
|Metrics and Metric groups||Metric is a key aspect of measuring verification effectiveness. It indicates test status, functional coverage, code coverage assertions etc.Metric groups can be used for hierarchy in the verification plan.|
|User defined properties of metrics||Enables to summarize results based on these properties. User can filter results with the help of expressions involving these properties and this enables users to quickly identify the root of the problem.|
|Consolidation of results by metric groups.||The ability of the tool to present data in a consolidated form enables users to understand a large amount of data.|
|A simple web based interface to view the plan and its associated results on a website||This helps the verification team share the plan and results on an intranet. It also enables the users to analyze the data on the website using charts.|
- The summary column shows the status of the tests, the error messages from the environment where these tests were run
- For a test there can be multiple results and whenever a result fails, the log files were referred directly from the tool through hyperlinks.
- Additional columns were added to the right of the summary column, for summarizing results according to the result properties.