Giving back to tech

Projects to test AI chat bots locally

Posted on Tue 02 May 2023 in AI, chat bots, Python, LLM • Tagged with AI, chat bots, Python, LLM, ChatAI, ColossalChat, LocalAI, Text generation web UI, MLC LLM

Many people is interested in trying AI (Artificial Intelligence) chat bots locally, but they don't know how to start. These are some projects that I've found that can be used to test different LLMs (Language Learning Models) and chat bots:

Text generation web UI. It tries to be the AUTOMATIC1111/stable-diffusion-webui of text generation. Based on Gradio, it provides a web interface to test different LLMs.
Serge. It's an interface based on Svelte as web framework and llama.cpp for running mmodels. It's dockerized. It also provides an API.
openplayground. Works with local models and remote APIs. Can be installed as a Python package or run with Docker. It's implemented as a Flask application with a React frontend.
ChatAI. It is a desktop application for Windows and Ubuntu developed in Python with PyQt. It uses a Google T5-Flan model and it is based on A. I. Runner, a framework to run AI models.
ColossalChat. ColossalAI provides a set of tools to develop deep learning models, and it includes ColossalChat, which is a chat bot based …

Visual Studio Code unable to connect

Posted on Fri 21 April 2023 in Visual Studio Code • Tagged with Visual Studio Code, VS Code

Some times, I get this error:

Unable to connect to VS Code server: Error in request.
Error: connect ENOENT [...]

I always have to search for the solution, so I decided to write it down here. From this issue, I found that running this solves the problem:

VSCODE_IPC_HOOK_CLI=$(lsof | grep $USER | grep vscode-ipc | awk '{print $(NF-1)}' | head -n 1)

Metrics per stage in k6

Posted on Fri 25 November 2022 in k6 • Tagged with k6

If you want to have custom metrics per each stage in an executor, you have to tag the stages (with tagWithCurrentStageIndex()) and add bogus thresholds for each metric that you want with the tag stage:i (being i the number of each stage). The bogus threshold has to be different for gauges (max>0) and rates (rate>=0).

This example file shows how to do it:

import http from 'k6/http';
import { tagWithCurrentStageIndex } from 'https://jslib.k6.io/k6-utils/1.3.0/index.js';

const stages = [
    { target: 1, duration: '10s' },
    { target: 5, duration: '10s' },
];

export const options = {
  scenarios: {
    contacts: {
      executor: 'ramping-arrival-rate',
      timeUnit: '1s',
      preAllocatedVUs: 10,
      maxVUs: 200,
      stages: stages,
    },
  },
  // Uncomment the next line if you want the count statistic
  // summaryTrendStats: ['avg', 'min', 'med', 'max', 'p(90)', 'p(95)', 'p(99)', 'count'],
  thresholds: {
    // Intentionally empty. We'll programatically define our bogus
    // thresholds (to generate the sub-metrics) below. In your real-world
    // load test, you can add any real threshoulds you want here.
  }
}

function addThreshold(thresholdName, threshold) {
    if (!options.thresholds[thresholdName]) {
        options.thresholds[thresholdName] = [];
    }

    // 'max>=0' is a …

Installing Tensorflow Serving in Amazon EC2 Linux

Posted on Fri 11 November 2022 in tensorflow, ec2 • Tagged with tensorflow

How to install and test Tensorflow Serving in a Amazon EC2 instance running Amazon EC2 Linux 2. We will use docker and we will serve a resnet image. These are the commands:

# Install docker
sudo yum update
sudo yum install docker
sudo usermod -a -G docker ec2-user
newgrp docker
sudo systemctl enable docker.service
sudo systemctl start docker.service

# Prepare the resnet model
rm -rf /tmp/resnet
wget https://tfhub.dev/tensorflow/resnet_50/classification/1?tf-hub-format=compressed -o resnet.tar.gz
mv 1?tf-hub-format=compressed resnet.tar.gz
mkdir -p /tmp/resnet/123
tar xvfz resnet.tar.gz -C /tmp/resnet/123/

# Create and run a docker image with tensorflow serving using the resnet model
docker run -d --name serving_base tensorflow/serving
docker cp /tmp/resnet serving_base:/models/resnet
docker commit --change "ENV MODEL_NAME resnet" serving_base $USER/resnet_serving
docker kill serving_base
docker rm serving_base
docker run -p 8500:8500 -p 8501:8501 -t $USER/resnet_serving &

To test that the serving works, you can run

sudo yum install git
git clone https://github.com/tensorflow/serving …

Using IPVS in kube-proxy with eksctl

Posted on Mon 20 June 2022 in kubernetes, eksctl, kube-proxy • Tagged with kubernetes, eksctl, kube-proxy

I have a kubernetes cluster launched with eksctl. I can get the configuration of kube-proxy with:

kubectl edit configmap kube-proxy-config -n kube-system

I see that the default configuration uses the iptables mode. In order to change it, the mode parameter has to be changed to ipvs and the scheduler parameter in the ipvs section, which is initially empty, has to be assigned one of these policies:

rr: round-robin
lc: least connection
dh: destination hashing
sh: source hashing
sed: shortest expected delay
nq: never queue

Notice that the corresponding kernel modules must be present in the working node. You can connect with ssh to the node and check with modules are loaded with:

lsmod | grep ip_vs

In order to apply the configuration, kube-proxy has to be restarted with this command:

kubectl rollout restart ds kube-proxy -n kube-system

I get this:

ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 176128  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          163840  8 xt_conntrack,nf_nat,xt_state,xt_nat,nf_conntrack_netlink,xt_connmark,xt_MASQUERADE,ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs

This means that the modules …

Older Posts Newer Posts