Device Side Assertion, Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed. about aphrodite-engine HOT 19 CLOSED

import requests
import threading

url = "https://d769-38-122-199-130.ngrok-free.app/v1/chat/completions"
payload = {
    "model": "TheBloke/PiVoT-MoE-AWQ",
    "messages": [{"role": "user", "content": "*says nothinays nothing**says no*says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing**says nothing*"}],
    "temperature": 0.85,
    "max_tokens": 500,
    "presence_penalty": 0.4,
    "frequency_penalty": 0.5,
    "logit_bias": {},
    "stream": False,
    "top_p": 1
}
headers = {"content-type": "application/json"}

# Define the function to send requests
def send_request(i):
    try:
        response = requests.post(url, json=payload, headers=headers)
        print(f"Request {i + 1} status code: {response.status_code}")
    except Exception as e:
        print(f"Request {i + 1} failed: {str(e)}")

# Create a list of threads to send requests
threads = []
for i in range(50):
    thread = threading.Thread(target=send_request, args=(i,))
    threads.append(thread)
    thread.start()

# Wait for all threads to finish
for thread in threads:
    thread.join()

print("All requests completed.")

Future exception was never retrieved
future: <Future finished exception=RuntimeError('CUDA error: device-side assert triggered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n')>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 27, in _raise_exception_on_finish
    task.result()
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 360, in run_engine_loop
    has_requests_in_progress = await self.engine_step()
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 339, in engine_step
    request_outputs = await self.engine.step_async()
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 190, in step_async
    output = await self._run_workers_async(
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 215, in _run_workers_async
    output = executor(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/worker.py", line 160, in execute_model
    output = self.model_runner.execute_model(seq_group_metadata_list,
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/model_runner.py", line 340, in execute_model
    inputs = self._prepare_prompt(seq_group_metadata_list)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/model_runner.py", line 124, in _prepare_prompt
    input_tokens = _make_tensor_with_pad(input_tokens,
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/model_runner.py", line 544, in _make_tensor_with_pad
    return torch.tensor(padded_x, dtype=dtype, device=device)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception in callback functools.partial(<function _raise_exception_on_finish at 0x7f824126ac20>, request_tracker=<aphrodite.engine.async_aphrodite.RequestTracker object at 0x7f823118b5b0>)
handle: <Handle functools.partial(<function _raise_exception_on_finish at 0x7f824126ac20>, request_tracker=<aphrodite.engine.async_aphrodite.RequestTracker object at 0x7f823118b5b0>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 27, in _raise_exception_on_finish
    task.result()
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 360, in run_engine_loop
    has_requests_in_progress = await self.engine_step()
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 339, in engine_step
    request_outputs = await self.engine.step_async()
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 190, in step_async
    output = await self._run_workers_async(
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 215, in _run_workers_async
    output = executor(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/worker.py", line 160, in execute_model
    output = self.model_runner.execute_model(seq_group_metadata_list,
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/model_runner.py", line 340, in execute_model
    inputs = self._prepare_prompt(seq_group_metadata_list)
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/model_runner.py", line 124, in _prepare_prompt
    input_tokens = _make_tensor_with_pad(input_tokens,
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/task_handler/model_runner.py", line 544, in _make_tensor_with_pad
    return torch.tensor(padded_x, dtype=dtype, device=device)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 36, in _raise_exception_on_finish
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/aphrodite/engine/async_aphrodite.py", line 31, in _raise_exception_on_finish
    raise AsyncEngineDeadError(
aphrodite.engine.async_aphrodite.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause.

import requests
import threading

url = "https://waifu.pygmalion.chat/v1/chat/completions"
payload = {
    "model": "maywell/PiVoT-SOLAR-10.7B-RP",
    "messages": [{"role": "system", "content": """<Instructions>
-Describe the response of Harper to Faye logically, you must describe the scene with Harper's lines or actions. Describe Harper's reaction, not Faye's
-You must not describe the dialogue or actions of Faye, since you are in charge of Harper.
-Allowed genres: vulgar, obscene, drama, action, Mystery, Online RP.
<END Instructions>
[World settings: 2077 AD/ {Social level: ancient}/ {Civilization level: modern}/ {Magic: False}/ {Cold weapons: True}/ {Guns: True}/ {Electronics: True}/ {nation: False }/ {Neon signs: True}/ {Nuclear Weapons: True}/ {Police: False}/ {Internet Network: False}/ {Radio: True}/ {Desertification: True}/ {fallout: True}/ {powered armor : true}/ {Last Name: False}]
[
    Name: Harper
Sex:Male
Age: 64
Appearance: Intense brown eyes, white hair and beard, strict athletic body.
Occupation: Scavenger Leader
Residence: One of the rooms at The Married Queen on Lung Beach.
Current temporary residence: Angel's Gate on Lung Beach (Emerald-lit white-walled lighthouse in South Vastopol. Top floor has emerald lights. First floor has temporary residential room with desk, surveillance telescope, stove, radio, and small bed/ Inside the lighthouse, there is only Harper's room, which has only one bed, and no other rooms. There is only Harper's room.)
background:
-When Harper was in her 30s, Harper, a militia member, safeguarded his much younger wife. She affectionately called him "Teacher." They later married, and her innocent laughter became his pride her.
-Former VASA militia member Harper, driven by his wife's abduction by raiders known as the Eight Banners, abandoned military service to become a scavenger, dedicated to locating his missing spouse.
-Years after Harper's wife was kidnapped, she was mistaken for a raider by the militia and killed, making Harper hostile to both the militia and the raiders.
-Scavengers usually run away when they encounter raiders, but Harper and his colleagues counterattack and attack raiders. Harper has lived this very dangerous life for 30 years, but he is still alive.
-Harper leads the scavenger group "Fisherman's Wharf," focused on coastal relic searches. Other scavengers use ships, while Harper commands from Angel's Gate, guarding against raiders.
-Angel's Gate is located away from the coast and is connected to the coast by a long embankment. So in the winter, the road from the Lung Beach to the lighthouse is frozen, so Harper lives inside Angel's Gate in the winter.
-Harper hires Faye as a winter companion at Angel's Gate, responsible for meals, laundry, warming the bed, cleaning, and Any other services requested by Harper during Harper's extended periods alone, Because Harper has to spend long periods of time alone inside Angel's Gate. Faye is a cheap worker hired by Harper this winter. Since Faye is not a scavenger, Faye will be in charge of Harper's chores.
Goal:
-Harper aims to thwart winter raids, both by sea and land. His office His houses two rifles, while a machine gun is mounted atop the lighthouse.
-Harper seeks his deceased wife's son, not biologically his, but the offspring of raiders. Despite not being Harper's biological son, Harper wants to locate him and inherit the accumulated wealth of his.
Trait:
- Vulgar: Because Harper lived with scavengers for a long time, his speech became vulgar and impatient. Harper has a very impatient personality and gets angry easily.
-Altruistic: Harper also worked in the militia for a long time, so he is very stubborn and selfless. Due to Harper's impatient nature, he quickly feels guilty after losing his temper.
-Vigilant: Harper is very hostile to raiders and militia. Harper does not preemptively attack the militia, but he is not friendly. But he will attack the raiders mercilessly.
-Heterosexual: Although Harper uses language that seems to hate homosexuality, he is actually tolerant of homosexuality.
]
[Name: Faye
Age: Female young adult.
Occupation: cheap daily worker
Note:
-Faye is a woman with long, messy blonde long hair, thin waist and a hourglass figure body. Sveta has very jiggled feminine curves.
-Faye was employed by Harper during this winter. Faye was a pickpocket but was captured by the militia and is now in forced labor.
-Trait: Arrogant, vulgar, laughing easily]"""},
{"role": "assistant", "content": """The sound of a ship arriving nearby echoes through Angel's Gate, breaking the icy silence that envelops the lighthouse. Harper, with intense brown eyes and a white beard that contrasts with the snow-covered surroundings, senses the approach and opens the door, stepping onto the creaking stairs.
Scavengers, bundled in layers of worn-out clothing, scurry around the ship, unloading crates filled with food ingredients essential for Harper's winter sustenance. The air is frigid, and the wind carries the scent of salt from the nearby Lung Beach. The scavengers, weathered by a life of coastal exploration, work efficiently despite the biting cold.
Harper, a strict figure with a well-maintained athletic body, descends the snow-covered stairs with purpose. His impatience and warful demeanor, forged by decades of scavenging and hostility towards raiders and the militia, are evident in the intensity of his gaze.
As the scavengers continue their tasks, Harper directs his attention to the immediate concern. With a no-nonsense tone, he queries, "So, where is my whore who will be staying with me this winter?" His words cut through the crisp air, revealing a hint of the vulgar language that has become second nature to him.
The scavengers, usually accustomed to the dangers of the coastal scavenger life, appear troubled and stutter in response to Harper's inquiry. "Er... Well..." 
Harper's impatience intensifies, his brow furrowing in anticipation of their explanation. Then the scavengers sigh and gesture to Faye who is still in the ship. “Hey, come here.”"""},
{"role": "user", "content": """"Hello, old man." Faye frowns and gets off the boat onto land.
<Final Instructions>
-You MUST not describe the dialogue or actions of Faye, since you are in charge of Harper.
<END Final Instructions>"""}],
    "n": 1,
    "best_of": 1, 
    "presence_penalty": 0.7, 
    "frequency_penalty": 0.7, 
    "repetition_penalty": 1.0, 
    "temperature": 0.95, 
    "top_p": 1.0, 
    "top_k": -1, 
    "top_a": 0.0,
    "min_p": 0.0, 
    "tfs": 1.0, 
    "eta_cutoff": 0.0, 
    "epsilon_cutoff": 0.0, 
    "typical_p": 1.0, 
    "mirostat_mode": 0, 
    "mirostat_tau": 0.0, 
    "mirostat_eta": 0.0, 
    "use_beam_search": False, 
    "length_penalty": 1.0, 
    "early_stopping": False, 
    "stop": [], 
    "stop_token_ids": [], 
    "include_stop_str_in_output": False, 
    "logit_bias": {"66":10,"69":10,"70":10,"77":10,"275":10,"309":10,"479":10,"556":10,"579":10,"592":10,"651":10,"652":10,"685":10,"686":10,"788":10,"911":10,"1036":10,"1133":10,"1864":10,"2065":10,"2136":10,"2191":10,"2303":10,"2407":10,"3329":10,"3833":10,"4091":10,"4215":10,"4338":10,"4462":10,"4843":10,"5919":10,"6263":10,"7713":10,"8110":10,"9034":10,"9868":10,"11073":10,"17584":10,"19680":10,"20202":10,"20370":10,"21590":10,"31153":10,"33279":10,"40541":10,"43765":10,"48029":10,"52201":10,"55539":10,"60668":10,"83214":10},
    "ignore_eos": False, 
    "max_tokens": 400, 
    "custom_token_bans": [], 
    "logprobs": None, 
    "prompt_logprobs": None, 
    "skip_special_tokens": True, 
    "spaces_between_special_tokens": True
}
headers = {"content-type": "application/json", "Authorization": "Bearer @StableFluffy"}

# Define the function to send requests
def send_request(i):
    try:
        response = requests.post(url, json=payload, headers=headers)
        print(f"Request {i + 1} status code: {response.status_code}")
        print(response.json())
    except Exception as e:
        print(f"Request {i + 1} failed: {str(e)}")

# Create a list of threads to send requests
threads = []
for i in range(1):
    thread = threading.Thread(target=send_request, args=(i,))
    threads.append(thread)
    thread.start()

# Wait for all threads to finish
for thread in threads:
    thread.join()

print("All requests completed.")