[TMLProject2-f1fd] Details ============================ Generated On: 2024-10-31 22:14:19 UTC TML Solution DAG Parameters' Details: User Chosen Parametets ---------------------------- STEP 1: Get TML Core Params: `tml_system_step_1_getparams_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - solutionname - TMLProject2-f1fd * - solutiontitle - My Solution Title * - solutiondescription - This is an awesome real-time solution built by TSS * - brokerhost - 127.0.0.1 * - brokerport - 9092 * - cloudusername - None * - ingestdatamethod - MQTT STEP 2: Create Kafka Topics: `tml_system_step_2_kafka_createtopic_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - companyname - Otics * - myname - Sebastian * - myemail - Sebastian.Maurice * - mylocation - Toronto * - replication - 1 * - numpartitions - 1 * - enabletls - 1 * - microserviceid - * - raw_data_topic - cisco-network-mainstream * - preprocess_data_topic - cisco-network-preprocess * - ml_data_topic - ml-data * - prediction_data_topic - prediction-data STEP 3: `Produce to Kafka Topics `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PRODUCETYPE - MQTT * - TOPIC - cisco-network-mainstream * - PORT - 44251 * - IDENTIFIER - MQTT Subscription Topic: tml/cybersecurity * - HTTPADDR - https:// * - FROMHOST - hammed-VMware-Virtual-Platform,127.0.1.1 * - TOHOST - 127.0.1.1 * - CLIENTPORT - 8883 * - TSS_CLIENTPORT - 8883 * - TML_CLIENTPORT - 8883 STEP 4: Preprocesing Data: `tml-system-step-4-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - cisco-network-mainstream * - preprocess_data_topic - cisco-network-preprocess * - preprocessconditions - * - delay - 70 * - array - 0 * - saveasarray - 1 * - topicid - -999 * - rawdataoutput - 1 * - asynctimeout - 120 * - timedelay - 0 * - preprocesstypes - min,max,trend,anomprob,variance,avg * - pathtotmlattrs - --pathtotmlattrs-- * - identifier - Detect potential cyber attacks and monitor network * - jsoncriteria - uid=hostName,filter:allrecords~subtopics=hostName,hostName,hostName~values=inboundpackets,outboundpackets,pingStatus~identifiers=inboundpackets,outboundpackets,pingStatus~datetime=lastUpdated~msgid=~latlong= STEP 4b: Preprocesing Data: `tml-system-step-4b-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - --raw_data_topic2-- * - preprocess_data_topic - --preprocess_data_topic2-- * - preprocessconditions - --preprocessconditions2-- * - delay - --delay2-- * - array - --array2-- * - saveasarray - --saveasarray2-- * - topicid - --topicid2-- * - rawdataoutput - --rawdataoutput2-- * - asynctimeout - --asynctimeout2-- * - timedelay - --timedelay2-- * - preprocesstypes - --preprocesstypes2-- * - pathtotmlattrs - --pathtotmlattrs2-- * - identifier - --identifier2-- * - jsoncriteria - --jsoncriteria2-- STEP 5: Entity Based Machine Learning : `tml-system-step-5-kafka-machine-learning-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - cisco-network-preprocess * - ml_data_topic - ml-data * - modelruns - --modelruns-- * - offset - -1 * - islogistic - --islogistic-- * - networktimeout - --networktimeout-- * - modelsearchtuner - --modelsearchtuner-- * - processlogic - --processlogic-- * - dependentvariable - --dependentvariable-- * - independentvariables - --independentvariables-- * - rollbackoffsets - --rollbackoffsets-- * - topicid - -999 * - consumefrom - iot-preprocess * - fullpathtotrainingdata - --fullpathtotrainingdata-- * - transformtype - --transformtype-- * - sendcoefto - --sendcoefto-- * - coeftoprocess - --coeftoprocess-- * - coefsubtopicnames - --coefsubtopicnames-- STEP 6: Entity Based Predictions: `tml-system-step-6-kafka-predictions-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - cisco-network-preprocess * - ml_prediction_topic - --ml_prediction_topic-- * - streamstojoin - --streamstojoin-- * - inputdata - --inputdata-- * - consumefrom - --consumefrom2-- * - offset - -1 * - delay - 70 * - usedeploy - --usedeploy-- * - networktimeout - --networktimeout-- * - maxrows - --maxrows-- * - topicid - -999 * - pathtoalgos - --pathtoalgos-- STEP 7: Real-Time Visualization: `tml-system-step-7-kafka-visualization-dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - vipervizport - 9005 * - topic - cisco-network-preprocess,cisco-network-privategpt * - dashboardhtml - tml-cisco-network-privategpt-monitor.html * - secure - 1 * - offset - -1 * - append - 0 * - chip - amd64 * - rollbackoffset - 400 STEP 8: `tml_system_step_8_deploy_solution_to_docker_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Docker Container - hooluwole/TMLProject2-f1fd-amd64 (https://hub.docker.com/r/hooluwole/TMLProject2-f1fd-amd64) * - Docker Run Command - docker run -d -p 39849:39849 -p 37139:37139 -p 56601:56601 -p 8883:8883 \-\-env TSS=0 \-\-env SOLUTIONNAME=TMLProject2-f1fd \-\-env SOLUTIONDAG=solution_preprocessing_ai_mqtt_dag-TMLProject2-f1fd \-\-env GITUSERNAME=Dynamo7001 \-\-env GITREPOURL=https://github.com/Dynamo7001/raspberrypi \-\-env SOLUTIONEXTERNALPORT=39849 \-\-env CHIP=amd64 \-\-env SOLUTIONAIRFLOWPORT=37139 \-\-env SOLUTIONVIPERVIZPORT=56601 \-\-env DOCKERUSERNAME='hooluwole' \-\-env CLIENTPORT=8883 \-\-env EXTERNALPORT=44251 \-\-env KAFKACLOUDUSERNAME='' \-\-env VIPERVIZPORT=9005 \-\-env MQTTUSERNAME='hammed' \-\-env AIRFLOWPORT=9000 \-\-env GITPASSWORD='' \-\-env KAFKACLOUDPASSWORD='' \-\-env MQTTPASSWORD='' \-\-env READTHEDOCS='' hooluwole/TMLProject2-f1fd-amd64 STEP 9: `tml_system_step_9_privategpt_qdrant_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PrivateGPT Container - maadsdocker/tml-privategpt-with-gpu-nvidia-amd64 * - PrivateGPT Run Command - docker run -d -p 8001:8001 --net=host --gpus all --env PORT=8001 --env GPU=1 --env COLLECTION=tml --env WEB_CONCURRENCY=1 --env CUDA_VISIBLE_DEVICES=0 maadsdocker/tml-privategpt-with-gpu-nvidia-amd64 * - Qdrant Container - qdrant/qdrant * - Qdrant Run Command - docker run -d -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage:z qdrant/qdrant * - Consumefrom - iot-preprocess * - pgpt_data_topic - cisco-network-privategpt * - offset - -1 * - rollbackoffset - 400 * - topicid - -999 * - enabletls - 1 * - partition - -1 * - prompt - Do the device data show any malfunction or defects? * - context - This is IoT data from devices. The data are anomaly probabilities for each IoT device. If voltage or current probabilities are low, it is likely the device is not working properly. * - jsonkeytogather - hyperprediction * - keyattribute - Voltage,current * - keyprocesstype - anomprob * - vectordbcollectionname - tml * - concurrency - 1 * - CUDA_VISIBLE_DEVICES - 0 * - pgpthost - http://127.0.0.1 * - pgptport - 8001 * - hyperbatch - _0 STEP 10: `tml_system_step_10_documentation_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Solution Documentation URL - https://TMLProject2-f1fd.readthedocs.io