python - Multiprocessing value() ctypes.c_wchar_p cause thread to stuck, alternative? - Stack Overflow

admin管理员组

文章数量:1122832

I'm doing concurrency using Multiprocessing() and Threading(). The idea is to spawn multiple threads from multiple process. The threads needs to access global variable that use Value() so that variable can be changed and and distributed across process (And threads I assume). So the structure is:

from multiprocessing import Queue, Process, Value
dc = {A: Queue(cpu_count), B: Value(ctypes.c_wchar_p, '', lock = False), C: Value('i', 0, lock = False)}
Process1 as supplement process that spawns worker threads:
Threads1 as task_producer for Queue
Threads2 giving flag signal for KeyboardInterrupt dc[c].value
Process3-8 as main process that spawns multiple threads:
Threads1-4 from Process3-8 access dc[b].value to connect

But once accessed, the threads stuck when loading the dc[b].value. But when printing dc[b], it works and mention the existence of Value():

c_wchar_p(2184070568752)

For the sake of the test, I've created some minimal reproducible examples:

from multiprocessing import Process, Queue, Manager, cpu_count, Value, Lock
import threading
import ctypes
import time

def print_props():
    while True:
        print('Process1 ', props['A'].value)

def change_props():
    while True:
        if props['A'].value == '0':
            props['A'].value = '1'
        else:
            props['A'].value = '0'
        print('Process2 ', props['A'].value)

def props_thread(n):
    global props
    props = n
    task_producer_thread = threading.Thread(target = change_props)
    task_producer_thread.start()
    task_producer_thread.join()

def main_thread(n):
    global props
    props = n
    task_producer_thread = threading.Thread(target = print_props)
    task_producer_thread.start()
    task_producer_thread.join()

def main():
    parent_process = []
    props_thread_process = Process(target = main_thread, args = (props, ))
    props_thread_process.start()
    parent_process.append(props_thread_process)
    
    main_thread_process = Process(target = props_thread, args = (props, ))
    main_thread_process.start()
    parent_process.append(main_thread_process)
    
    for i in parent_process:
        i.join()

if __name__ == '__main__':
    global props
    props = {"A": Value(ctypes.c_wchar_p, '0', lock = False), "B": Value('i', 0, lock = False)}
    main()

It looks like Value(ctypes.c_wchar_p, '', lock = False) can't be accessed simultaneously accross threads and process, but using Value('i', 0, lock = False) which corresponds for c_ulong(0), work just fine without stuck.

I'm expecting that Threading() inside a Multiprocessing() could also access Value(). So, when calling dc[b].value, it returns the domain string. Is there a way to distribute editable global variable string accross multiprocessing and threads?

Running latest python3.12 on Windows 11 22H2

I'm doing concurrency using Multiprocessing() and Threading(). The idea is to spawn multiple threads from multiple process. The threads needs to access global variable that use Value() so that variable can be changed and and distributed across process (And threads I assume). So the structure is:

from multiprocessing import Queue, Process, Value
dc = {A: Queue(cpu_count), B: Value(ctypes.c_wchar_p, '', lock = False), C: Value('i', 0, lock = False)}
Process1 as supplement process that spawns worker threads:
Threads1 as task_producer for Queue
Threads2 giving flag signal for KeyboardInterrupt dc[c].value
Process3-8 as main process that spawns multiple threads:
Threads1-4 from Process3-8 access dc[b].value to connect

But once accessed, the threads stuck when loading the dc[b].value. But when printing dc[b], it works and mention the existence of Value():

c_wchar_p(2184070568752)

For the sake of the test, I've created some minimal reproducible examples:

from multiprocessing import Process, Queue, Manager, cpu_count, Value, Lock
import threading
import ctypes
import time

def print_props():
    while True:
        print('Process1 ', props['A'].value)

def change_props():
    while True:
        if props['A'].value == '0':
            props['A'].value = '1'
        else:
            props['A'].value = '0'
        print('Process2 ', props['A'].value)

def props_thread(n):
    global props
    props = n
    task_producer_thread = threading.Thread(target = change_props)
    task_producer_thread.start()
    task_producer_thread.join()

def main_thread(n):
    global props
    props = n
    task_producer_thread = threading.Thread(target = print_props)
    task_producer_thread.start()
    task_producer_thread.join()

def main():
    parent_process = []
    props_thread_process = Process(target = main_thread, args = (props, ))
    props_thread_process.start()
    parent_process.append(props_thread_process)
    
    main_thread_process = Process(target = props_thread, args = (props, ))
    main_thread_process.start()
    parent_process.append(main_thread_process)
    
    for i in parent_process:
        i.join()

if __name__ == '__main__':
    global props
    props = {"A": Value(ctypes.c_wchar_p, '0', lock = False), "B": Value('i', 0, lock = False)}
    main()

It looks like Value(ctypes.c_wchar_p, '', lock = False) can't be accessed simultaneously accross threads and process, but using Value('i', 0, lock = False) which corresponds for c_ulong(0), work just fine without stuck.

I'm expecting that Threading() inside a Multiprocessing() could also access Value(). So, when calling dc[b].value, it returns the domain string. Is there a way to distribute editable global variable string accross multiprocessing and threads?

Running latest python3.12 on Windows 11 22H2

• python
• python-3.x
• python-multiprocessing
• python-multithreading

Share Improve this question Follow edited Nov 23, 2024 at 6:49 Guild asked Nov 23, 2024 at 4:56 GuildGuild 1133 bronze badges

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 0

Is there a way to distribute editable global variable string across multiprocessing and threads?

You gave few details about your code; the OP doesn't offer a repex.

Across threads is straightforward. Though you probably want to protect access with a mutex.

Across processes implies writing to a file descriptor, such as a pipe or an mmap'd file. Easiest thing to do would be to append lines to a log file, or INSERT rows in a relational database.

[ Answers ]

1. The alternative to ctypes.c_wchar_p string is to use ctypes.c_char bytes-string. The downside is you need to .decode() and .encode() each time you access the value.

2. Use multiprocessing.Array('c', b'bytes-string') to replace Value() for string shared memory. Using Value() will raise in an exception "bytes is too long"

[ Summary ]

As summary for future self:

1. Use multiprocessing.Array() for shared string
2. Use multiprocessing.Value() for shared int
3. Use multiprocessing.shared_memory.ShareableList() for shared list
4. Use multiprocessing.Manager() for shared memory as Proxy, unrecommended because it's slow.

本文标签： pythonMultiprocessing value() ctypescwcharp cause thread to stuckalternativeStack Overflow