To manage expectations, here are some things to consider when using idTV's framework.

Token Limits and Context Windows

Because idTV is a collection of multiple agents, it is important to consider the cost of inference for running a stream.

While text generation models have become relatively affordable, generating images, audio, and video can be costly when running a continuous stream. We recommend carefully monitoring token usage and managing costs. For cost efficiency, we suggest running open source models with publicly available weights on dedicated GPU cluster.

Limitations of Local Models

Local models can be a great way to run streams cheaply and efficiently. However, they do have limitations. For agents that require large context windows (e.g., parsing through large amounts of data), local models may not be optimal. Additionally, for non-text-based generation tasks, local models may struggle to produce consistent results.