[AINews] RWKV "Eagle" v5: Your move, Mamba • ButtondownTwitterTwitter

The discussions in this section provide insights into various AI Discord communities and highlight the key points from platforms such as TheBloke, Nous Research AI, OpenAI, and LM Studio. The content ranges from debates on model origins and effective training sample sizes to advancements in AI models such as Eagle 7B and Mixtral kernels. Additionally, the summaries touch on topics like RoPE theta settings, decoding dilemmas, and tokenizer nuances in large language models. This section also covers community dialogues on GPU choices for LLM tasks, beta bugs in LM Studio releases, and challenges with hardware configurations and Linux GPU acceleration. Overall, the summaries reflect the vibrant exchange of knowledge and experiences within these AI-focused communities.

Flash-Attention Adaptation for Jax:

Discussions about porting flash-attention to Jax are foregrounded by challenges related to dependencies on PyTorch with CUDA, leading to considerations for forking and modifying the original repository to accommodate Jax bindings. This adaptation aims to manage the compatibility issues between torch-cuda and jax-cuda due to cuda version conflicts.

T5 Models Missing Flash Attention:

Concerns have been raised about the absence of flash attention implementation in T5 models, marking it as a significant gap in leveraging this technology within that particular framework.

AI Expert Opinions Versus Media Representations:

The discordance between AI experts' insights and media portrayals, particularly involving figures like Gary Marcus, sparks debate over the impact of academic rivalries and media misrepresentations on public understanding. This discussion highlights the Gell-Mann Amnesia effect and the challenges of conveying accurate AI advancements.

Existential Risks and Silicon Valley Preppers:

A diverging conversation emerges around existential risks and the culture of prepping, underscored by skepticism towards the motivations behind such activities. Yet, evidence points toward high-profile figures like Larry Page and Mark Zuckerberg investing in secluded refuges, stirring a complex dialogue on readiness versus skepticism toward catastrophic events.

Seeking 2023 News Datasets for Model Training:

The demand for up-to-date news datasets for model training in 2023 and possibly January 2024 is evident, with current resources like the common crawl dump from December being deemed unsuitable due to its unfiltered nature. Suggestions for alternatives, like scraping PROQUEST, indicate a proactive search for viable datasets.

Embedding Strategies and Activation Mechanisms in LLMs Discussed:

A rich dialogue happens around the transition from tying to untying word embeddings in large language models (LLMs), activation beacon mechanisms for maintaining information over long sequences, and the investigation of post-training sparsification techniques like SliceGPT. This discussion is enlivened by critiques of current benchmarks' construction, notably the MMMU benchmark, and shows a growing interest in self-play research projects.

Language Model Evaluation Harness Insights:

Tweaks to the LM evaluation harness, including seed changes and the incorporation of the RWKV library, highlight an ongoing effort to assess language models consistently. These adjustments, alongside discussions about per example metrics and the repetition penalty's impact, stress the community’s dedication to refining evaluation strategies.

GPT-NeoX Development Hurdles and Solutions:

Efforts to address GPT-NeoX developmental challenges, such as Apex build troubles and multi-node deployment obstacles, illustrate a communal commitment to making the tool more accessible and efficient across various architectures. The notion of creating an opinionated Apex fork and setting up a build pipeline for scalability and ease points toward proactive solutions for future-proofing and wider architecture support.

TheBloke ▷ #general (1395 messages🔥🔥🔥):

• Debate Over Miqu's Origin: Ongoing speculation about whether miqu-1-70b is a leaked Mistral Medium or a fine-tuned Llama 2 model.

• Analysis and Benchmarks Shared: Users shared analyses comparing miqu to models like Mistral and RWKV, showing mixed results especially at different bit quantizations.

• Performance Discussions on Various Hardware: Discussion on using hardware like M2 Ultra, Macbook M3 Max, and GPU rigs featuring Nvidia's A100 and 4090 cards for running AI models.

• TabbyAPI and Model Running Challenges: Challenges and techniques discussed for running models using tools like TabbyAPI, llama.cpp, and exl2.

• Discussion Over RWKV and New Model Developments: Updates and capabilities of RWKV models mentioned, with tools and projects designed to enhance model accessibility and performance.

Links mentioned:

• Chat with Open Large Language Models
• 🦅 Eagle 7B : Soaring past Transformers with 1 Trillion Tokens Across 100+ Languages
• How to play with the Raven model?
• Tweet from nisten (@nisten)
• Panchovix/goliath-120b-exl2 · Hugging Face
• miqudev/miqu-1-70b at main
• alpindale/miqu-1-70b-pytorch · Hugging Face
• miqudev/miqu-1-70b · Hugging Face
• Dial Up Internet - Sound Effect (HD)

Deepseek's System Prompt Magic:

.ben.com shared a template for a system prompt for the Deepseek Coder model, focusing on handling default and provided system prompts for consistency. More details can be found in the discussion.

Counterfactual Prompting for Aligning LLMs:

Gabriel_syme introduced a paper on counterfactual prompting to align large language models' response styles without human intervention. The paper explores enhancing models' generation styles innately (Download PDF).

Exploring Infinite Context Scaling in LLMs:

A discussion started by euclaise about a paper proposing a novel approach enabling infinite context scaling in large language models drew mixed reactions. While beneficial for roleplay and chat agents, concerns about facts retention were raised (Study more here).

Exllamav2 Enhancements and GitHub Release:

.ben.com discussed the advantages of Exllamav2 for Large Language Models (LLMs), including a 2x throughput increase on a 3090ti GPU and the release of an OpenAI API compatible LLM inference server based on Exllamav2 on GitHub.

Eagle 7B's Remarkable Achievement:

Nonameusr highlighted the launch of Eagle 7B, a 7.52B parameter model on the RWKV-v5 architecture. It outperforms other 7B class models in multi-lingual benchmarks and rivals top-tier model performance in English evaluations, with significantly lower inference cost (Find out more).

Links mentioned:

• 🦅 Eagle 7B : Soaring past Transformers with 1 Trillion Tokens Across 100+ Languages
• Aligning Large Language Models with Counterfactual DPO
• Tweet from BlinkDL (@BlinkDL_AI)
• internlm/internlm2-math-20b · Hugging Face
• exllamav2/examples/multiple_caches.py at master · turboderp/exllamav2
• GitHub - bjj/exllamav2-openai-server: An OpenAI API compatible LLM inference server based on ExLlamaV2.
• Reddit - Dive into anything
• GitHub - TryForefront/tuna

OpenAI ▷ #gpt-4-discussions (75 messages🔥🔥):

• Users discussed issues with GPT-4's retrieval capability from custom knowledge bases and technical problems with creating Custom GPTs.

• The new @-calling feature in GPT-4 for leveraging multiple models in a conversation was explored.

• Instances of GPT-4 providing unsatisfactory responses and advising Bing searches were reported, prompting investigations into user experience.

• Confusions around GPT model switching persistence were clarified, highlighting potential misunderstandings and flexibility in interacting with multiple models.

OpenAI ▷ #prompt-engineering (299 messages🔥🔥):

• Discussions included rule reminders, exploration of prompt variable types, and the effectiveness of emotion prompts in automated prompt engineering.

• Participants shared self-critique techniques and technical queries regarding prompt engineering.

• Resources like OpenAI's Prompt Engineering Guide were mentioned to aid in learning and troubleshooting.

OpenAI ▷ #api-discussions (299 messages🔥🔥):

• Rule clarifications regarding posting URLs and discussions on prompt variables were prevalent in this section.

• Users explored SudoLang and EmotionPrompt for automated prompt engineering.

• Discussions on model quantization and quantization-aware training were presented in these conversations.

• Hardware recommendations, including GPUs and CPUs for running large language models, were among the topics of interest.

LM Studio ▷ #💬-general (316 messages🔥🔥):

• Discussions ranged from uncensored text adventure models and new model discoveries to troubleshooting and optimization tips.

• Members expressed interest in integrating vision models into LM Studio for text generation.

• Anticipation for future multimodal models and developing more versatile tools for AI-driven content creation were highlighted.

LM Studio ▷ #🎛-hardware-discussion (144 messages🔥🔥):

• Topics included VRAM requirements, CPU considerations, and hardware recommendations for running LLMs efficiently.

• Discussions on quantization of large language models and preferences for mixing different GPU brands were prevalent.

• Community resources like LLM leaderboards and arXiv for research updates were shared.

LM Studio ▷ #🧠-feedback (5 messages):

• Users reported errors in LM Studio related to AVX2 instructions, iCloud Drive, and unknown error codes, seeking troubleshooting assistance.

• Discussions revolved around hardware limitations for running LLMs effectively.

LM Studio ▷ #🧪-beta-releases-chat (6 messages):

• Updates on yi-vl support and troubleshooting GPU acceleration issues on Linux were discussed in this section.

• Beta release fixes for Windows users were welcomed, emphasizing improvements for OpenCL compatibility.

LM Studio ▷ #autogen (7 messages):

• Autogen Studio errors and troubleshooting, including issues with config files and error handling, were shared among users.

• Suggestions for exploring the NexusRaven-V2 GitHub repository for function calling were mentioned.

Flash-Attention Jax Porting Woes:

• @nshepperd contemplates porting flash-attention to Jax due to dependencies on pytorch with cuda, suggesting forking the repo.

The Absence of Flash Attention in T5 Models:

• @rallio expresses surprise at the lack of flash attention for T5 models.

AI Expertise and Media Misinterpretations:

• @exirae discusses how AI experts like Gary Marcus in media can dilute public understanding of AI's capabilities.

Concerns Around X-Risks and Prepping Culture:

• Users debate existential risks, prepping culture, and high-profile individuals investing in refuges.

Searching for Current News Datasets for Model Training:

• @danfosing seeks news datasets from 2023-2024, highlighting the scarcity of quality recent news datasets and suggesting alternatives like PROQUEST.

Links Mentioned:

• Inside Mark Zuckerberg’s Top-Secret Hawaii Compound
• Reclusive Google co-founder Larry Page hiding out in Fiji

Self-Research vs. Spoonfeeding Information:

A debate ensued between @mrdragonfox and @vhariational regarding whether directly providing answers or encouraging self-research benefits the questioner more. @mrdragonfox expressed a preference for encouraging self-research to improve problem-solving skills.

Plans to Open Source Discussed Project:

In response to a query about the availability of a project on GitHub, @hugoduprez mentioned plans to open source the project and promised to keep the community updated.

Announcement of Arithmo2-Mistral-7B Model:

@ajindal introduced the Arithmo2-Mistral-7B model which shows improvement on GSM8K, GSM8K PoT, and MATH benchmarks over its predecessor. Links to the model and LoRA adapter are shared on Hugging Face and detailed information can be found on the project's GitHub page.

This section discusses various topics related to Colab compute quandaries, Clip Retrieval tools, the release of WhiteRabbitNeo-33B-v1 model, cybersecurity insights, and an off-topic inquiry about the OpenAI framework. The users express confusion over Colab compute as a Pro user and seek a reliable Clip Retrieval tool. The release of the WhiteRabbitNeo-33B-v1 model by Migel Tissera is highlighted, along with a Twitter Space discussion on cybersecurity. Additionally, there is an inquiry about the OpenAI framework without specific details.

• Users on the platform are engaging in debates over the effectiveness of Bing's and Google's AI integrations, showcasing varying opinions on their search capabilities.
• Efforts to enhance AI model training and fine-tuning are being discussed, indicating the community's commitment to advancing AI image synthesis and model sophistication.
• Google's Bard surpassing GPT-4 and a new AI model excelling in text-to-image tasks are highlighted in the ongoing discussions.
• Insights into the challenges and advancements in Reinforcement Learning from Human Feedback strategies are being shared.
• Eagle 7B, a notable 7.52B parameter model, is introduced for its efficiency and multilingual proficiency.
• Perplexity AI users are discussing various topics, including model versions, subscription model queries, and technical support accessibility.
• Perplexity AI users are also sharing their positive experiences with the AI tool in specific searches, coding learning, and discovering information on healthy berries.
• LlamaIndex discussions include building enterprise RAG systems, challenges for AI engineers, and enhancing RAG with knowledge graphs.
• Latent Space discussions cover topics like building RAG systems, AI photography learning, and critiques of AI search engine experiences.
• A variety of topics are being discussed in different channels, from API issues to PDF parsing solutions and leveraging retrievers for search applications.

DiscoResearch 📌 #embedding_dev (1 messages):

• sebastian.bodza: >80k

DiscoResearch 📌 #discolm_german (10 messages🔥):

• In Search of the Optimal DiscoLM Setup with Ollama: User @jannikstdl asked the community for advice on integrating DiscoLM German with Ollama, focusing on finding the most effective modelfile configuration.
• Template Troubles Lead to Lackluster LLM Responses: @jannikstdl shared their initial template code for Ollama which resulted in the LLM only responding with

Alignment Lab AI 📌 #general-chat (1 messages):

• Seeking 2023 News Datasets: @danfosing is looking for datasets that include news articles from 2023 and possibly January 2024. They also mentioned an inability to post in another specific channel (<#1117625732189933650>).

Alignment Lab AI 📌 #oo (5 messages):

• Dedicated DM Grind: autometa mentioned they've sent like 10 DMs to a recipient, emphasizing their commitment to the "grind".
• In Search of Missing Discussions: ilovescience inquired if discussions were happening elsewhere, with teknium confirming that no, discussions were not occurring in another location. This was succinctly followed by a solitary emoji from teknium, indicating perhaps a light-hearted acknowledgment of the situation.