[gen][fuzz] impl backend constraints and fuzzing loop generation exp safety

README.md

@@ -10,17 +10,18 @@
 
 ## Backend-Model Support
 
-| Backend\Model | ONNX/PyTorch | TensorFlow |
-| ------------- | ------------ | ---------- |
-| TVM           | ✅            |            |
-| ONNXRuntime   | ✅            |            |
-| TensorRT      | ✅            |            |
-| TFLite        |              | ⚠️          |
-| XLA           |              | ⚠️          |
-| IREE          |              | ⚠️          |
+<center>
+
+| Model\Engine | [TVM](https://github.com/apache/tvm) | [ORT](https://github.com/microsoft/onnxruntime) | [TensorRT](https://github.com/NVIDIA/TensorRT) | [TFLite](https://www.tensorflow.org/lite) | [XLA](https://www.tensorflow.org/xla) | [IREE](https://github.com/iree-org/iree) |
+| ------------ | ------------------------------------ | ----------------------------------------------- | ---------------------------------------------- | ----------------------------------------- | ------------------------------------- | ---------------------------------------- |
+| PyTorch-ONNX | ✅                                    | ✅                                               | ✅                                              |                                           |                                       |                                          |
+| TensorFlow   |                                      |                                                 |                                                | ⚠️                                         | ⚠️                                     | ⚠️                                        |
+
 
 ✅: Supported; ⚠️: Beta support; Others are not supported yet -- Contributions are welcome!
 
+</center>
+
 ## Setup
 
 **Install latest stable release:**
@@ -135,108 +136,3 @@ pytest tests/tensorflow -s
 ## Paper
 
 Our paper is accepted by ASPLOS'23 and the pre-print is now available on [![arXiv](https://img.shields.io/badge/arXiv-2207.13066-b31b1b.svg)](https://arxiv.org/abs/2207.13066).
-
-
-
-<!--
-### Coverage Evaluation
-
-**WIP: Scripts under `experiments/` are not ready yet due to recent refactors.**
-
-To run coverage evaluation, first compile the DL framework with LLVM's [source-based code coverage](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html). The commands below should be at least compatible with LLVM-14.
-
-<details><summary><b>NNSmith</b> <i>[click to expand]</i></summary>
-<div>
-
-```shell
-bash experiments/cov_exp.sh
-python experiments/cov_merge.py -f nnsmith-tvm-* nnsmith-ort-*  # generate merged_cov.pkl
-```
-
-</div>
-</details>
-
-
-<details><summary><b>LEMON</b> <i>[click to expand]</i></summary>
-<div>
-
-Please prepare 100GB disk space to store LEMON's outputs.
-
-```shell
-# step 1: Run LEMON to generate models (https://github.com/ganler/LEMON);
-# step 2:
-# For TVM
-python experiments/lemon_tf2onnx.py --lemon_output_dir /PATH/TO/LEMON/lemon_outputs/ --onnx_dir lemon-onnx
-python experiments/cov_eval.py --model_dir lemon-onnx    \
-                               --report_folder lemon-tvm \
-                               --backend tvm --lib '../tvm/build/libtvm.so ../tvm/build/libtvm_runtime.so' \
-                               --llvm-version 14 # if you compile tvm w/ llvm 14 instrumented on ubuntu.
-# For ORT:
-python experiments/cov_eval.py --model_dir lemon-onnx \
-                               --report_folder lemon-ort \
-                               --backend ort \
-                               --lib '../onnxruntime/build/Linux/RelWithDebInfo/libonnxruntime_providers_shared.so ../onnxruntime/build/Linux/RelWithDebInfo/libonnxruntime.so' \
-                               --llvm-version 14
-python experiments/cov_merge.py -f lemon-tvm lemon-ort # generate merged_cov.pkl
-```
-
-</div>
-</details>
-
-<details><summary><b>GraphFuzzer</b> <i>[click to expand]</i></summary>
-<div>
-
-*The original [paper](https://conf.researchr.org/details/icse-2021/icse-2021-papers/68/Graph-based-Fuzz-Testing-for-Deep-Learning-Inference-Engines) does not give it a name so we call it GraphFuzzer for convenience.*
-
-```shell
-# Make sure ORT dtype support config file is generated.
-python nnsmith/dtype_test.py --cache config/ort_cpu_dtype.pkl
-
-# TVM
-python experiments/graphfuzz.py --time_budget 14400 --onnx_dir /PATH/TO/LEMON/graphfuzz-tvm-onnx
-python experiments/cov_eval.py --model_dir /PATH/TO/LEMON/graphfuzz-tvm-onnx    \
-                               --report_folder graphfuzz-tvm \
-                               --backend tvm --lib '../tvm/build/libtvm.so ../tvm/build/libtvm_runtime.so' \
-                               --llvm-version 14
-
-# ORT
-python experiments/graphfuzz.py --time_budget 14400 --onnx_dir /PATH/TO/LEMON/graphfuzz-ort-onnx --ort_cache config/ort_cpu_dtype.pkl
-python experiments/cov_eval.py --model_dir /PATH/TO/LEMON/graphfuzz-ort-onnx \
-                               --report_folder graphfuzz-ort \
-                               --backend ort \
-                               --lib '../onnxruntime/build/Linux/RelWithDebInfo/libonnxruntime_providers_shared.so ../onnxruntime/build/Linux/RelWithDebInfo/libonnxruntime.so' \
-                               --llvm-version 14
-
-python experiments/cov_merge.py -f graphfuzz-tvm graphfuzz-ort # generate merged_cov.pkl
-```
-
-</div>
-</details>
-
-<details><summary><b>Visualization</b> <i>[click to expand]</i></summary>
-<div>
-
-```shell
-mkdir results # Store those files in results
-# TVM coverage.
-python experiments/viz_merged_cov.py --folders lemon-tvm graphfuzz-tvm nnsmith-tvm --tvm --pdf --tags 'LEMON' 'GraphFuzzer' 'NNSmith' --venn --output main_result
-# ORT coverage.
-python experiments/viz_merged_cov.py --folders lemon-ort graphfuzz-ort nnsmith-ort --ort --pdf --tags 'LEMON' 'GraphFuzzer' 'NNSmith' --venn --output main_result
-```
-
-</div>
-</details>
-
-### Evaluate input searching algorithm
-
-```shell
-# Run experiments.
-bash experiments/input_search_exp.sh 10
-bash experiments/input_search_exp.sh 20
-bash experiments/input_search_exp.sh 30
-
-# visualization
-python experiments/plot_inp_search_merge.py --root 512-model-10-node-exp \
-                                                   512-model-20-node-exp \
-                                                   512-model-30-node-exp
-``` -->

doc/log-and-err.md

@@ -21,11 +21,9 @@ ${NNSMITH_CMD} hydra.verbose=fuzz
 ${NNSMITH_CMD} hydra.verbose="[fuzz,exec]"
 ```
 
-#### Logging things into file
+#### Where the log is?
 
-By default, NNSmith logs things in `console` mode where loggings will only be flushed to STDIO.
-
-To log outputs into a file, add flag `hydra/job_logging=file`. The log file will be in [`${hydra.runtime.output_dir}/${hydra.job.name}.log`](https://hydra.cc/docs/1.2/tutorials/basic/running_your_app/working_directory/) (e.g., `output/${DATE}/${JOB_ID}/${APP}.log`).
+By default, NNSmith logs things both in `console` and `file`. You can find the loggings in [`outputs/${DATE}/${APP}.log`](https://hydra.cc/docs/1.2/tutorials/basic/running_your_app/working_directory/) (current working directory).
 
 ## Errors
 

experiments/README.md

@@ -0,0 +1,4 @@
+**NOTE**: Scripts under this folder are out of maintaince that it cannot work with the latest implementation.
+
+However, it can work with `620645967a14d6a7b077cedd9c2c03ed74af50d9` which is used in result evaluation of our ASPLOS'22.
+Commits later than `0239c066ff94ac5307bfd5924e678d9d214cafbc` should be able to render the output results in a way I did in the paper.

nnsmith/abstract/extension.py

@@ -1,20 +1,11 @@
 import functools
 import types
-from typing import List, Type
+from typing import List
 
 from nnsmith.abstract.op import AbsOpBase
 from nnsmith.abstract.tensor import AbsTensor
 
-TABLE_UPDATE_REQUIRES = {}
-
-
-def change_requires(
-    system_name: str, op_types: List[Type[AbsOpBase]]
-) -> List[Type[AbsOpBase]]:
-    for optype in op_types:
-        if optype.name() in TABLE_UPDATE_REQUIRES[system_name]:
-            optype.requires = TABLE_UPDATE_REQUIRES[system_name][optype.name()]
-    return op_types
+BACKEND_REQUIRES = {}
 
 
 def copy_requires(f):
@@ -30,7 +21,7 @@ def __init__(self, tag: str, opname: str):
         self.opname = opname
 
     def __call__(self, f):
-        TABLE_UPDATE_REQUIRES.setdefault(self.tag, {}).setdefault(self.opname, f)
+        BACKEND_REQUIRES.setdefault(self.tag, {}).setdefault(self.opname, f)
         return f
 
 
@@ -46,7 +37,7 @@ def patch_with_prev(op: AbsOpBase, itensors: List[AbsTensor]):
                 self.prev_fn = copy_requires(op.requires)
             return f(op, itensors) + self.prev_fn(op, itensors)
 
-        TABLE_UPDATE_REQUIRES.setdefault(self.tag, {}).setdefault(
+        BACKEND_REQUIRES.setdefault(self.tag, {}).setdefault(
             self.opname, patch_with_prev
         )
         return patch_with_prev

nnsmith/backends/factory.py

@@ -1,11 +1,13 @@
 import sys
 import traceback
 from abc import ABC, abstractmethod
-from typing import Callable, Dict, List, Optional, Union
+from typing import Callable, Dict, List, Optional, Type, Union
 
 import numpy as np
 
 from nnsmith.abstract.dtype import DType
+from nnsmith.abstract.extension import BACKEND_REQUIRES
+from nnsmith.abstract.op import AbsOpBase
 from nnsmith.abstract.tensor import AbsTensor
 from nnsmith.difftest import assert_allclose
 from nnsmith.error import InternalError
@@ -248,3 +250,11 @@ def init(name, target="cpu", optmax=True, catch_process_crash=False, **kwargs):
             )
         else:
             raise ValueError(f"unknown backend: {name}")
+
+    def add_constraints(self, op_types: List[Type[AbsOpBase]]) -> List[Type[AbsOpBase]]:
+
+        for optype in op_types:
+            if optype.name() in BACKEND_REQUIRES[self.system_name]:
+                optype.requires = BACKEND_REQUIRES[self.system_name][optype.name()]
+
+        return op_types

nnsmith/backends/tensorrt.py

@@ -145,5 +145,5 @@ def skip_dtypes(cls) -> List[DType]:
 
 
 @patch_requires(TRTFactory.system_name, "core.Pool2d")
-def RulePool2d(self: AbsOpBase, _: List[AbsTensor]) -> List[Union[z3.ExprRef, bool]]:
+def RulePool2d(self: AbsOpBase, _: List[AbsTensor]) -> List[Union[z3.BoolRef, bool]]:
     return [nnsmith_lt(nnsmith_mul(self.kernel_h_size, self.kernel_w_size), 10000)]

nnsmith/cli/dtype_test.py

@@ -3,7 +3,7 @@
 
 from nnsmith.backends import BackendFactory
 from nnsmith.materialize import Model
-from nnsmith.narrow_spec import load_topset_from_auto_cache
+from nnsmith.narrow_spec import auto_opconfig
 
 
 @hydra.main(version_base=None, config_path="../config", config_name="main")
@@ -18,7 +18,7 @@ def main(cfg: DictConfig):
     else:
         factory = None
     model_type = Model.init(cfg["model"]["type"], backend_target=backend_cfg["target"])
-    load_topset_from_auto_cache(model_type, factory)
+    auto_opconfig(model_type, factory)
 
 
 if __name__ == "__main__":

nnsmith/cli/fuzz.py

@@ -1,21 +1,23 @@
 import random
 import time
+import traceback
 from pathlib import Path
 
 import hydra
 from omegaconf import DictConfig
 
 from nnsmith.backends.factory import BackendFactory
 from nnsmith.cli.model_exec import verify_testcase
+from nnsmith.error import InternalError
 from nnsmith.graph_gen import concretize_graph, random_model_gen
 from nnsmith.logging import FUZZ_LOG
 from nnsmith.macro import NNSMITH_BUG_PATTERN_TOKEN
 from nnsmith.materialize import Model, Schedule, TestCase
-from nnsmith.narrow_spec import opset_from_auto_cache
+from nnsmith.narrow_spec import auto_opset
 from nnsmith.util import mkdir, set_seed
 
 
-class Reporter:
+class StatusCollect:
     def __init__(self, root):
         self.root = Path(root)
         mkdir(self.root)
@@ -33,7 +35,7 @@ def __init__(
     ):
         self.cfg = cfg
 
-        self.reporter = Reporter(cfg["fuzz"]["root"])
+        self.status = StatusCollect(cfg["fuzz"]["root"])
 
         self.factory = BackendFactory.init(
             cfg["backend"]["type"],
@@ -47,7 +49,7 @@ def __init__(
             model_cfg["type"], backend_target=cfg["backend"]["target"]
         )
         self.ModelType.add_seed_setter()
-        self.opset = opset_from_auto_cache(self.ModelType, self.factory)
+        self.opset = auto_opset(self.ModelType, self.factory)
 
         seed = cfg["fuzz"]["seed"] or random.getrandbits(32)
         set_seed(seed)
@@ -102,9 +104,9 @@ def validate_and_report(self, testcase: TestCase) -> bool:
             self.cfg["cmp"],
             factory=self.factory,
             testcase=testcase,
-            output_dir=self.reporter.get_next_bug_path(),
+            output_dir=self.status.get_next_bug_path(),
         ):
-            self.reporter.n_bugs += 1
+            self.status.n_bugs += 1
             return False
         return True
 
@@ -113,10 +115,26 @@ def run(self):
         while time.time() - start_time < self.timeout_s:
             seed = random.getrandbits(32)
             FUZZ_LOG.debug(f"Making testcase with seed: {seed}")
-            testcase = self.make_testcase(seed)
+            try:
+                testcase = self.make_testcase(seed)
+            except InternalError as e:
+                raise e  # propagate internal errors
+            except Exception:
+                FUZZ_LOG.error(
+                    f"`make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., {self.cfg['model']['type']})."
+                )
+                FUZZ_LOG.error(traceback.format_exc())
+                repro = "nnsmith.model_gen"
+                repro += f" mgen.seed={seed}"
+                repro += f" mgen.max_nodes={self.cfg['mgen']['max_nodes']}"
+                repro += f" model.type={self.cfg['model']['type']}"
+                repro += f" backend.target={self.cfg['backend']['target']}"
+                FUZZ_LOG.error(f"repro with: {repro}")
+                continue
+
             if not self.validate_and_report(testcase):
                 FUZZ_LOG.warning(f"Failed model seed: {seed}")
-            self.reporter.n_testcases += 1
+            self.status.n_testcases += 1
 
 
 @hydra.main(version_base=None, config_path="../config", config_name="main")

nnsmith/cli/model_gen.py

@@ -9,7 +9,7 @@
 from nnsmith.graph_gen import concretize_graph, random_model_gen, viz
 from nnsmith.logging import MGEN_LOG
 from nnsmith.materialize import Model, Schedule, TestCase
-from nnsmith.narrow_spec import opset_from_auto_cache
+from nnsmith.narrow_spec import auto_opset
 from nnsmith.util import mkdir
 
 
@@ -39,7 +39,7 @@ def main(cfg: DictConfig):
         factory = None
 
     gen = random_model_gen(
-        opset=opset_from_auto_cache(ModelType, factory),
+        opset=auto_opset(ModelType, factory),
         init_rank=mgen_cfg["init_rank"],
         seed=seed,
         max_nodes=mgen_cfg["max_nodes"],

nnsmith/config/main.yaml

@@ -70,5 +70,5 @@ cmp:
   save: null # path to save the bug report if `bug_presence` is "report"
 
 defaults:
-  - override hydra/job_logging: console
+  - override hydra/job_logging: file
   - override hydra/hydra_logging: colorlog